A method, apparatus, device and medium for deploying a large model agent

Abstract

The application discloses a large model agent deployment method and device, equipment and medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: after receiving an agent deployment request, determining agent configuration information of a large model agent to be deployed, and determining a target large model and model metadata; determining the available resource situation of a pre-designed computing resource pool, and performing query matching according to the available resource situation and the model metadata to determine a plurality of hardware-compatible candidate computing resource nodes; performing a load prediction operation on the large model agent to be deployed after being put into operation to obtain a predicted load, and generating a target deployment decision scheme based on the predicted load, a preset scheduling strategy, the available resource situation and the candidate computing resource nodes; adjusting the runtime parameters of the target large model according to the resource characteristics of the target computing resource node, and loading the adjusted target large model on the target computing resource node to deploy and start an agent instance of the large model agent to be deployed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a large model agent deployment method and device, equipment and medium. BACKGROUND

[0002] With the rapid development of large model (such as large language model, multi-modal model, etc.) technology, the number of agent applications (such as intelligent customer service, content creation assistant, data analysis assistant, etc.) based on these models is increasing. These agent applications usually require a large amount of computing resources (especially special accelerators such as GPUs (Graphics Processing Unit)) and memory to run, and the demand for resources may change dynamically over time. At the same time, different agents may need to call different versions and types of large models, and these models have specific requirements for hardware resources (such as GPU models and memory sizes). Current large model agent deployment mainly relies on two types of technology: general resource scheduling systems and traditional model deployment tools. The resource scheduling system only supports static resource allocation, which is difficult to adapt to fluctuations in agent load, requires manual labeling of node tags, often leads to idle or insufficient resources, and has low deployment efficiency. The model deployment tool only focuses on model inference service, and the deployment, scaling and capacity expansion of agent instances often require manual intervention or rely on complex scripts, which is slow in response and difficult to meet the needs of peak periods of business, and is difficult to maintain. At the same time, since large models usually have specific requirements for hardware (such as specific GPU architecture and memory size), manually selecting the appropriate model version and deployment environment is time-consuming and prone to errors.

[0003] In summary, how to optimize the deployment method of large model agents to improve the efficiency, stability and resource utilization of large model agent services is a technical problem that needs to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a large model agent deployment method, device, equipment and medium, which can optimize the deployment method of large model agents to improve the efficiency, stability and resource utilization of large model agent services. The specific solutions are as follows:

[0005] In a first aspect, the present application provides a large model agent deployment method, comprising:

[0006] After receiving the agent deployment request, the agent configuration information of the large model agent to be deployed is determined, and the corresponding target large model and the corresponding model metadata are determined according to the agent configuration information; the agent configuration information includes a preset scheduling strategy configured;

[0007] determine available resource conditions of the pre-designed computing resource pool, and perform query matching on the pre-designed computing resource pool according to the available resource conditions and the model metadata, to determine a plurality of hardware-compatible candidate computing resource nodes;

[0008] perform a load prediction operation on the to-be-deployed large model agent after it is online, to obtain a corresponding predicted load, and generate a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource conditions, and the candidate computing resource nodes; the target deployment decision scheme includes a target computing resource node determined from the candidate computing resource nodes;

[0009] adjust runtime parameters of the target large model according to resource characteristics of the target computing resource node, to obtain an adjusted target large model, and load the adjusted target large model on the target computing resource node according to the target deployment decision scheme, to deploy and start an agent instance of the to-be-deployed large model agent.

[0010] Optionally, before determining the agent configuration information of the to-be-deployed large model agent after receiving the agent deployment request, the method further includes:

[0011] automatically discovering a plurality of heterogeneous computing resources currently available, and deploying a resource collection agent to a plurality of computing resource nodes corresponding to the plurality of heterogeneous computing resources, to collect resource information of the plurality of computing resource nodes respectively through the resource collection agent;

[0012] abstracting the resource information into corresponding computing resource labels, and dividing according to features of the resource information, to generate the pre-designed computing resource pool for managing the plurality of computing resource nodes.

[0013] Optionally, the load prediction operation on the to-be-deployed large model agent after it is online, to obtain a corresponding predicted load, includes:

[0014] obtaining historical monitoring data of the to-be-deployed large model agent;

[0015] using a time series analysis method or a target machine learning method, predicting a load change trend in a preset time period after the to-be-deployed large model agent is online based on the historical monitoring data, to obtain a corresponding predicted load;

[0016] Correspondingly, in the process of generating a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource conditions, and the candidate computing resource nodes, the process includes:

[0017] Pre-allocate or pre-reserve the candidate computing resource node according to the predicted load, and determine a target computing resource node most suitable for the target large model from the candidate computing resource node based on the predicted load, the preset scheduling strategy and the available resource condition.

[0018] Optionally, the generating a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource condition and the candidate computing resource node comprises:

[0019] If the preset scheduling strategy is a first scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node for the purpose of minimizing cost.

[0020] If the preset scheduling strategy is a second scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node for the purpose of optimizing performance.

[0021] If the preset scheduling strategy is a third scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node for the purpose of maximizing resource utilization.

[0022] If the preset scheduling strategy is a fourth scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node for the purpose of minimizing response time.

[0023] Optionally, the adjusting a runtime parameter of the target large model according to the resource characteristics of the target computing resource node comprises:

[0024] Determining model memory requirement data in the model metadata and available memory data of the target computing resource node;

[0025] Calculating a preset memory occupancy formula using the model memory requirement data and the available memory data to quantify hardware compatibility of the target large model and the target computing resource node, and obtaining a corresponding quantization result;

[0026] Adjusting a batch size of the target large model according to the quantization result.

[0027] Optionally, after deploying and starting the agent instance of the to-be-deployed large model agent, the method further comprises:

[0028] The agent instance is continuously monitored to collect its running status information in real time, and a health check is performed on the agent instance based on the running status information to determine whether the agent instance is running normally and to obtain the corresponding judgment result.

[0029] If the judgment result indicates that the agent instance is not running normally, then an alarm operation is performed on the agent instance, and the agent instance is restarted or replaced.

[0030] Accordingly, the step of performing a health check on the agent instance based on the running status information to determine whether the agent instance is running normally and obtaining the corresponding judgment result includes:

[0031] The performance indicators in the running status information are compared with preset performance indicator thresholds to obtain the corresponding comparison results.

[0032] If the comparison result indicates that the performance index is higher than the preset performance index threshold, then the performance index is determined to be abnormal, and the result of the judgment that the agent instance is not running normally is obtained.

[0033] Optionally, after deploying and starting the agent instance of the large model agent to be deployed, the method further includes:

[0034] Continuously monitor the agent instances to obtain corresponding monitoring results;

[0035] Adjust the resource allocation of the agent instance based on the monitoring results, or adjust the scheduling weight of the agent instance, or perform horizontal scaling operations on the agent instance, or update the model configuration of the target large model.

[0036] Secondly, this application provides a deployment apparatus for a large-scale intelligent agent, comprising:

[0037] The data determination module is used to determine the agent configuration information of the large model agent to be deployed after receiving the agent deployment request, and to determine the corresponding target large model and corresponding model metadata based on the agent configuration information; the agent configuration information includes the configured preset scheduling strategy;

[0038] The resource pool query module is used to determine the available resources of a preset computing resource pool, and to query and match the preset computing resource pool based on the available resources and the model metadata to determine several hardware-compatible candidate computing resource nodes.

[0039] The load prediction module is configured to perform an online load prediction operation on the to-be-deployed large model agent to obtain a corresponding predicted load, and generate a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource condition and the candidate computing resource node, wherein the target deployment decision scheme includes a target computing resource node determined from the candidate computing resource node.

[0040] The large model loading module is configured to adjust runtime parameters of the target large model according to resource characteristics of the target computing resource node to obtain an adjusted target large model, and load the adjusted target large model on the target computing resource node according to the target deployment decision scheme to deploy and start an agent instance of the to-be-deployed large model agent.

[0041] In a third aspect, the present application provides an electronic device, comprising:

[0042] A memory configured to save a computer program;

[0043] A processor configured to execute the computer program to implement the large model agent deployment method.

[0044] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program; wherein the computer program is executed by a processor to implement the large model agent deployment method.

[0045] In the present application, after receiving an agent deployment request, the agent configuration information of the to-be-deployed large model agent is determined, and the corresponding target large model and the corresponding model metadata are determined according to the agent configuration information; the agent configuration information includes a configured preset scheduling strategy; the available resource situation of a pre-designed computing resource pool is determined, and the pre-designed computing resource pool is queried and matched according to the available resource situation and the model metadata, to determine a plurality of hardware-compatible candidate computing resource nodes; a load prediction operation is performed on the to-be-deployed large model agent after being put into operation, to obtain a corresponding predicted load, and a target deployment decision scheme is generated based on the predicted load, the preset scheduling strategy, the available resource situation, and the candidate computing resource nodes; the target deployment decision scheme includes a target computing resource node determined from the candidate computing resource nodes; the runtime parameters of the target large model are adjusted according to the resource characteristics of the target computing resource node, to obtain an adjusted target large model, and the adjusted target large model is loaded on the target computing resource node according to the target deployment decision scheme, to deploy and start an agent instance of the to-be-deployed large model agent; the agent instance is continuously monitored, to perform scheduling management on the agent instance according to the obtained monitoring result. As can be seen from the above, after receiving an agent deployment request, the present application first determines the configuration information (including a preset scheduling strategy) of the to-be-deployed large model agent, and determines the target large model and the model metadata accordingly, then queries the available resource situation of the pre-designed computing resource pool, and matches the hardware-compatible candidate computing resource nodes in the pre-designed computing resource pool in combination with the model metadata, then predicts the predicted load of the to-be-deployed large model agent after being put into operation, generates a deployment decision scheme including a target computing resource node based on the predicted load, the preset scheduling strategy, the available resource situation, and the candidate computing resource nodes, then adjusts the runtime parameters of the target large model according to the resource characteristics of the target computing resource node, loads the adjusted model on the target computing resource node to deploy and start an agent instance, and finally continuously monitors the instance and performs scheduling management according to the monitoring result. In this way, through the above process of the present application, from the early configuration matching to the resource screening, and then to the load prediction and scheme generation, it can be ensured that the deployment decision is suitable for the agent demand and the actual resource situation, and the overall utilization rate of the heterogeneous computing resources is improved; by adjusting the model runtime parameters to adapt to the target node, the model running efficiency can be improved; the hardware affinity matching is used to determine a plurality of hardware-compatible candidate computing resource nodes, which can dynamically screen compatible resources, reducing the complexity and error rate of manual configuration; continuous monitoring and scheduling management can respond to resource fluctuations or performance problems in instance running in real time, ensuring that the large model agent provides services stably and efficiently, and thus optimizing the deployment method of the large model agent to improve the efficiency, stability, and resource utilization rate of the large model agent service. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only only the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0047] Figure 1 A large model agent deployment method flowchart disclosed by the present application;

[0048] Figure 2 A system initialization and resource pool management flowchart disclosed by the present application;

[0049] Figure 3 An implementation architecture diagram of a dynamic resource pool and model adaptation system disclosed by the present application;

[0050] Figure 4 A specific large model agent deployment method flowchart disclosed by the present application;

[0051] Figure 5 A large model agent deployment device structure schematic diagram disclosed by the present application;

[0052] Figure 6 An electronic device structure diagram disclosed by the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] Current large model agent deployment mainly relies on two types of technologies: general resource scheduling system and traditional model deployment tool. The resource scheduling system only supports static resource allocation, which is difficult to adapt to the fluctuation of agent load, and needs manual annotation of node labels, often leading to idle or insufficient resources, low deployment efficiency; the model deployment tool only focuses on model inference service, and the deployment, expansion and contraction of agent instances often need manual intervention or rely on complex scripts, slow response speed, difficult to meet the demand of business peak period, and difficult to maintain. At the same time, since the large model usually has specific requirements for hardware (such as specific GPU architecture, memory size), manual selection of appropriate model version and deployment environment is time-consuming and prone to errors.

[0055] To overcome the above technical problems, the present application provides a large model agent deployment method, which can optimize the large model agent deployment method to improve the efficiency, stability and resource utilization of the large model agent service.

[0056] Referring to Figure 1 The embodiment of the present application discloses a large model agent deployment method, which comprises:

[0057] Step S11, after receiving the agent deployment request, determine the agent configuration information of the large model agent to be deployed, and determine the corresponding target large model and the corresponding model metadata according to the agent configuration information; the agent configuration information includes the configured preset scheduling strategy.

[0058] In this embodiment, after receiving the agent deployment request, the agent configuration information of the large model agent to be deployed containing the preset scheduling strategy is determined first, and then the corresponding target large model and the corresponding model metadata are determined according to the agent configuration information. Among them, the large model agent to be deployed includes but is not limited to intelligent customer service assistant, intelligent writing assistant, content creation assistant, data analysis assistant and other agent applications; the target large model includes but is not limited to large language model, multi-modal model, etc.

[0059] It should be pointed out that before deploying the large model agent, a pre-designed computing resource pool for managing available computing resources needs to be constructed, such as Figure 2A system initialization and resource pool management process diagram provided by the present application is shown, and the processing flow is as follows: automatically discovering a plurality of heterogeneous computing resources currently available, and deploying a resource collection agent to a plurality of computing resource nodes corresponding to the plurality of heterogeneous computing resources to collect resource information of each of the plurality of computing resource nodes through the resource collection agent; abstracting the resource information into corresponding computing resource labels, and dividing according to the characteristics of the resource information to generate the pre-designed computing resource pool for managing the plurality of computing resource nodes. Wherein, the resource information includes hardware specifications, real-time load states; the hardware specifications include but are not limited to model, computing power, video memory size, bandwidth, etc.; the real-time load state includes but is not limited to CPU (Central Processing Unit) usage, memory occupancy, GPU utilization, temperature, etc. That is, first, a plurality of heterogeneous computing resources available in the current network are automatically or manually discovered, such as different models of NVIDIA A100 GPU servers, V100 GPU servers, CPU servers, etc. Then, the resource collection agent is deployed to a plurality of computing resource nodes corresponding to the resources, and the resource information of each node is collected through the resource collection agent. Then, the resource information is uniformly abstracted into standardized computing resource labels such as "GPU-A100-40GB", "GPU-V100-16GB", "CPU-8 core", etc., and divided according to the characteristics of the resource information, so as to abstract the physical or virtual computing resources into standardized computing units (such as divided according to GPU model, video memory size, CPU core number, etc.), and finally generate a unified, dynamically expandable pre-designed computing resource pool pool-A for managing the plurality of computing resource nodes, hiding the differences of the underlying hardware, for subsequent scheduling.

[0060] It needs to be further pointed out that the present application also maintains a model warehouse for storing model metadata of the target large model registered by the user. The model metadata includes the minimum hardware specifications required for each model version (such as GPU model, minimum video memory), recommended hardware specifications, model file size, memory occupancy estimation during inference, etc. For example, the developer can register the LLM (i.e. Large Language Model) model v2.1 used by the "smart writing assistant" to the model warehouse, and record its metadata: minimum NVIDIA A100 GPU, video memory 24GB or more, recommended A100 40GB; model file size 80GB; peak memory occupancy during inference about 20GB. When the user needs to deploy an agent, the configuration of the agent can be defined, including the required called large model (specified version), processing logic, resource requirements (minimum / maximum number of GPUs, video memory size, CPU core number, memory size), network configuration, scaling strategy, etc., to initiate the agent deployment request. For example, the user creates a "smart customer service assistant" agent in the management interface, and configures it as follows: use model "LLM-Chat-3.5", specify version v1.2; resource requirements: minimum 1 GPU-A100-40GB, maximum 2; memory at least 64GB; scaling strategy: automatically scale when CPU utilization exceeds 80% for a continuous period of time.

[0061] It needs to be pointed out that, as Figure 3An implementation architecture diagram of a dynamic resource pool and model adaptation system provided by the application is shown. It includes a resource pool management module, a model adaptation module, an agent instance management module, a monitoring and prediction module, and a scheduling decision module. The resource pool management module is used to dynamically manage a heterogeneous computing resource pool, including resource discovery, access, abstraction, pooling, and dynamic scheduling and allocation according to agent requirements and prediction results, considering future load prediction results. The model adaptation module is used to match the affinity of models and hardware according to the large model and its version requested by the large model agent, combined with the available resource information provided by the resource pool management module, and dynamically configure the running parameters of the model based on the specific hardware matched. The agent instance management module is used to manage the life cycle of the large model agent instance, including deployment, start, stop, restart, destroy, scheduling, scaling, and state monitoring, supporting operation according to a preset strategy or manual instruction, and routing requests to appropriate instances according to the instructions of the scheduling decision module. The monitoring and prediction module is used to monitor resource usage and agent instance performance indicators in real time, and predict future load changes. The scheduling decision module is used to issue resource adjustment, instance scaling, or model configuration update decision instructions to the resource pool management module, agent instance management module, or model adaptation module based on the output of the monitoring and prediction module and a preset strategy. It can be understood that the key events and state changes generated by each module will be recorded for auditing and problem troubleshooting. In this way, the embodiment first specifies agent configuration information to provide accurate basis for subsequent selection of target large models, avoiding mismatch between selected models and deployment requirements; determines the preset scheduling strategy and model metadata in advance to reduce the blindness in the deployment process and lay the foundation for the deployment of large model agents; automatic resource discovery and collection can quickly cover various types of heterogeneous computing resources, avoiding the omission and inefficiency of manual troubleshooting; abstracting resource information into labels and constructing a resource pool by classification can make the properties of computing resources clearer, quickly locate resources that meet the requirements, and improve the efficiency of resource scheduling and management.

[0062] In step S12, the available resource conditions of the pre-designed computing resource pool are determined, and the pre-designed computing resource pool is queried and matched according to the available resource conditions and the model metadata to determine a plurality of hardware-compatible candidate computing resource nodes.

[0063] In this embodiment, the available resource situation of the pre-designed computing resource pool is first determined, and then the pre-designed computing resource pool is queried and matched in combination with the hardware requirements in the model metadata of the target large model, and finally a plurality of hardware compatible candidate computing resource nodes are determined. For example, a model requiring a large amount of video memory and high computing accuracy is preferentially matched to a high-performance GPU with sufficient video memory. When the hardware requirement is A100 40GB, the pre-designed computing resource pool is queried, and it is found that there are currently 2 A100 40GB servers (nodes A and B) and 1 V100 16GB server (node C), so nodes A and B are compatible hardware resources. In this way, the available situation of the resource pool is combined with the model metadata for matching in this embodiment, which can accurately screen out nodes that meet the hardware conditions for running large models, avoiding running failures caused by deploying models on hardware incompatible nodes; the candidate nodes are directly screened from the resource pool, without the need to check all computing resources one by one, greatly improving the efficiency of resource matching.

[0064] Step S13, performing a load prediction operation on the to-be-deployed large model agent after online, to obtain a corresponding predicted load, and generating a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource situation and the candidate computing resource node; the target deployment decision scheme includes a target computing resource node determined from the candidate computing resource node.

[0065] In this embodiment, the load prediction of the to-be-deployed large model agent after online is performed to obtain a predicted load, and then a target deployment decision scheme including a target computing resource node determined from the candidate node is generated in combination with the predicted load, the preset scheduling strategy, the available resource situation and the candidate computing resource node.

[0066] It should be noted that the load prediction operation after the online of the to-be-deployed large model agent is as follows: obtaining the historical monitoring data of the to-be-deployed large model agent; using a time series analysis method or a target machine learning method, predicting the load change trend of the to-be-deployed large model agent within a preset time period after being online based on the historical monitoring data to obtain a corresponding predicted load; accordingly, in the process of generating a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource condition, and the candidate computing resource node, the candidate computing resource node is pre-allocated or reserved according to the predicted load, and the target computing resource node most suitable for the target large model is determined from the candidate computing resource node based on the predicted load, the preset scheduling strategy, and the available resource condition. That is, first, the historical monitoring data (such as resource utilization and request volume) of the to-be-deployed large model agent is obtained, and then a time series analysis or target machine learning method is used to predict the resource demand and load change trend within a preset time period (such as several minutes to several hours) after being online based on the historical monitoring data to obtain a predicted load. When generating a target deployment decision scheme, the candidate computing resource node is pre-allocated or reserved according to the predicted load, the resource amount allocated can be dynamically adjusted according to the instance load (such as increasing / decreasing the number of allocated GPUs or the proportion of video memory), and the target computing resource node most suitable for the target large model is determined from the candidate node based on the predicted load, the demand of the to-be-deployed large model agent, the preset scheduling strategy, and the available resource condition, that is, the hardware closest to the recommended configuration of the model (such as A100 is better than V100). For example, when deploying an "intelligent customer service assistant" instance, according to its configuration (1 GPU-A100-40GB is required) and the state of the resource pool pool-A (nodes A and B are idle), combined with the prediction that the future traffic will rise, node A is selected to allocate 1 A100 40GB resource to the instance.

[0067] It needs to be further pointed out that the process flow of generating the target deployment decision scheme is as follows: if the preset scheduling strategy is the first scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node with the purpose of minimizing the cost; if the preset scheduling strategy is the second scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node with the purpose of optimizing the performance; if the preset scheduling strategy is the third scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node with the purpose of maximizing the resource utilization rate; if the preset scheduling strategy is the fourth scheduling strategy, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node with the purpose of minimizing the response time. That is, when generating the target deployment decision scheme, the different purposes are determined according to the different preset scheduling strategies: if it is the first scheduling strategy, the purpose is to minimize the cost; if it is the second scheduling strategy, the purpose is to optimize the performance; if it is the third scheduling strategy, the purpose is to maximize the resource utilization rate; and if it is the fourth scheduling strategy, the purpose is to minimize the response time. A corresponding target deployment decision scheme is generated based on the predicted load, the available resource condition and the candidate computing resource node. In addition, the preset scheduling strategy can also be a load balancing scheduling strategy. In this way, the embodiment generates a deployment scheme based on multi-dimensional information, can avoid the one-sidedness of the decision caused by relying on a single factor, and ensures that the deployment meets the overall resource management rules in combination with the scheduling strategy; the load prediction mode based on historical data and professional algorithms can more accurately predict the resource demand of the agent after going online, and avoid resource waste or deficiency caused by load estimation deviation; in combination with the predicted load for node pre-allocation or reservation and adaptive screening, it can be ensured that the target node is not only hardware compatible, but also can stably undertake the running load of the agent within a preset time period, and at the same time meets the scheduling strategy and the resource status, improves the efficiency and stability of the agent deployment; the deployment decision mode of different optimization targets according to the scheduling strategy can flexibly adapt to different deployment demand scenarios and screen the matching candidate computing resource nodes.

[0068] In step S14, the runtime parameters of the target large model are adjusted according to the resource characteristics of the target computing resource node to obtain an adjusted target large model, and the adjusted target large model is loaded on the target computing resource node according to the target deployment decision scheme to deploy and start the agent instance of the to-be-deployed large model agent.

[0069] In this embodiment, first, the runtime parameters of the target large model are dynamically adjusted according to the resource characteristics (such as the actual available amount of GPU memory and computing power) of the target computing resource node, such as batch size, precision, optimizer parameters, etc., to obtain the best performance on the current hardware, obtain the adjusted target large model, and then load the adjusted target large model on the target computing resource node according to the target deployment decision scheme, and finally complete the instance deployment and startup of the to-be-deployed large model agent.

[0070] It should be noted that the process of adjusting the runtime parameters of the target large model is as follows: determine the model memory requirement data in the model metadata, and determine the available memory data of the target computing resource node; use the model memory requirement data and the available memory data to calculate a preset memory occupation formula to quantify the hardware compatibility of the target large model and the target computing resource node, and obtain the corresponding quantization result; and adjust the batch size of the target large model according to the quantization result. That is, first, determine the model memory requirement data in the model metadata, such as LLM v2.1 requiring 24GB of memory, and the available memory data of the target computing resource node, such as the remaining memory of node A being 38GB, and then use these two types of data to calculate the preset memory occupation formula to dynamically quantify the hardware compatibility of the target large model and the target computing resource node and obtain the quantization result. The preset memory occupation formula is as follows:

[0071] M = k * B + C;

[0072] Wherein, M represents the total memory required for model running; k represents the memory required for each sample (token), that is, the additional memory required for processing each token after the model is loaded; B is the batch size, that is, the number of samples processed at a time; and C represents the basic memory required for loading the model onto the GPU (including model weights, intermediate results of runtime, etc. Fixed overhead). For example, querying the model metadata shows that k is 2GB / sample and C is 20GB, then the maximum batch size of node A (available memory 38GB) is calculated to be 9, and finally the batch size of the target large model is adjusted according to the quantization result, combined with the delay requirement (such as B being lower than 12), and finally B = 9 is set and written into the deployment configuration. For example, after determining that the target computing resource node is node A, the batch size of the model is automatically adjusted from the default 8 to 12 according to the actual available memory of 38GB (considering system overhead) to improve throughput while ensuring that it does not exceed the memory limit.

[0073] It should be noted that after the agent instance of the to-be-deployed large model agent is deployed and started, it can be monitored and managed in real time, and the processing flow is as follows: the agent instance is continuously monitored to collect the running state information of the agent instance in real time, and the agent instance is health checked based on the running state information to determine whether the agent instance is normally running, and a corresponding judgment result is obtained; if the judgment result represents that the agent instance is not normally running, an alarm operation is performed on the agent instance, and the agent instance is restarted or replaced; correspondingly, the processing flow of health checking the agent instance based on the running state information is as follows: the performance indicators in the running state information are compared with the preset performance indicator threshold to obtain a corresponding comparison result; if the comparison result represents that the performance indicators are higher than the preset performance indicator threshold, it is determined that the performance indicators are abnormal, and the judgment result that the agent instance is not normally running is obtained. The running state information includes resource utilization (CPU, memory, GPU utilization, video memory occupation), request processing delay, error rate, throughput, instance performance indicator data, etc., and can also include API (Application Programming Interface, application programming interface) call logs collected through the agent instance or API interface, the API call logs include call time, API Key, AppID, request parameter, response state, time consumption, number of consumed tokens, etc. That is, the agent instance is continuously monitored to collect its running state information in real time, and health checking is performed based on the running state information to determine whether the agent instance is normally running, and specifically, the performance indicators in the running state information are compared with the preset threshold, if the performance indicators are higher than the threshold (such as continuous high GPU utilization and high error rate), it is determined that the performance indicators are abnormal, and the result that the instance is not normally running is obtained. If it is determined that the instance is not normally running, an alarm is triggered and the agent instance is restarted or replaced.

[0074] It needs to be further pointed out that the operation of scheduling management of the agent instance includes multiple operations, and the processing flow is as follows: continuously monitoring the agent instance to obtain a corresponding monitoring result; adjusting the resource allocation of the agent instance according to the monitoring result, or adjusting the scheduling weight of the agent instance, or performing horizontal scaling operation on the agent instance, or updating the model configuration of the target large model. That is, after obtaining the monitoring result by continuously monitoring the agent instance, multiple management operations are taken based on the monitoring result, including adjusting the resource allocation of the instance (increasing / decreasing the allocation of a certain resource), adjusting the scheduling weight of the instance, performing horizontal scaling operation on the instance, or updating the model configuration of the target large model, and updating the system state after the management operation is executed. For example, considering the predicted load (increased by 50%), the current resource pool state (node B has available A100 40GB), the instance configuration (maximum 2 GPUs), and the preset policy (performance priority), the decision is to restart an "intelligent customer service assistant" instance to cope with the growth, and instruct the load balancing to distribute new requests to the two instances, while updating the system state to be expanded. In this way, the present embodiment adjusts the model runtime parameter according to the node resource characteristics, which can make the target large model better adapt to the hardware conditions of the node, avoid model running lag or failure caused by mismatch between parameters and resources; sequentially loading the model according to the deployment scheme and starting the instance can ensure that the deployment process is controllable, so that the agent instance can quickly and stably enter the available state; adjusting the batch size according to the formula quantification of the display memory compatibility can accurately match the model display memory demand and the available display memory of the node, while ensuring the stable operation of the model, and optimizing its running efficiency as much as possible, so that the target large model realizes the balance between performance and stability on the target node; the health check method of real-time monitoring and threshold comparison can quickly find the performance abnormality of the agent instance, and avoid the long-term continuous influence of the abnormal state on the service; the alarm combined with the restart or replacement processing mechanism can respond to faults in time, maximize the influence of instance exceptions on business, and ensure the stability and continuity of the large model agent service; the dynamic management method based on real-time monitoring results can make the agent instance always adapt to the actual running demand, while flexibly adjusting the scheduling weight and the model configuration, which can further optimize the instance running efficiency and service quality, and ensure that the large model agent provides stable and efficient services for a long time.

[0075] As can be seen from the above, after receiving the agent deployment request, the embodiment of the application first determines the configuration information (including the preset scheduling strategy) of the to-be-deployed large model agent, and determines the target large model and model metadata according to the configuration information, then queries the available resource situation of the pre-designed computing resource pool, matches the candidate computing resource nodes compatible with the hardware in the pre-designed computing resource pool in combination with the model metadata, predicts the predicted load after the to-be-deployed large model agent goes online, generates a deployment decision scheme including a target computing resource node based on the predicted load, the preset scheduling strategy, the available resource situation and the candidate computing resource node, then adjusts the runtime parameter of the target large model according to the resource characteristics of the target computing resource node, loads the adjusted model on the target computing resource node to deploy and start the agent instance, and finally continuously monitors the instance and performs scheduling management according to the monitoring result. In this way, through the above process of the embodiment of the application, from the early configuration matching to the resource screening, and then to the load prediction and scheme generation, it can ensure that the deployment decision is suitable for the agent demand and the actual resource situation, and improves the overall utilization rate of the heterogeneous computing resources; by adjusting the model runtime parameter to adapt to the target node, the model running efficiency can be improved; the hardware affinity matching is used to determine a plurality of hardware compatible candidate computing resource nodes, which can dynamically screen compatible resources, reduce the complexity and error rate of manual configuration; continuous monitoring and scheduling management can respond to resource fluctuations or performance problems in the instance running in real time, ensure that the large model agent provides services stably and efficiently, and further optimize the deployment method of the large model agent to improve the efficiency, stability and resource utilization rate of the large model agent service.

[0076] Based on the above embodiment, the application discloses a large model agent deployment method, which can optimize the deployment method of the large model agent to improve the efficiency, stability and resource utilization rate of the large model agent service. Next, taking the deployment of a new agent based on a specific large model as an example, assuming that a developer wants to deploy a new agent named "intelligent customer service assistant", the agent will use a large language model (for example, model-A) to process user inquiries, and will be aimed at Figure 4 The deployment method of the large model agent will be described in detail.

[0077] First, monitoring and metrics collection. The system continuously monitors the resource usage of existing agent instances (e.g., other deployed customer service or analysis agents), such as GPU utilization, memory occupancy, API call frequency, response latency, etc. At the same time, it collects the current state of the resource pools (e.g., previously created pool-A and pool-B), such as the number of remaining type-A GPUs from manufacturer-A and type-B GPUs from manufacturer-B in pool-A. The system may also collect historical data for future load prediction.

[0078] Then, load prediction and resource requirement analysis. Based on the collected metrics (such as historical API call volume and current queued request number), the system predicts the load level that the "intelligent customer service assistant" agent may reach in the initial stage (e.g., expected to handle 50 requests per second). Combined with the known resource consumption characteristics of model-A (e.g., inference delay, peak GPU memory occupancy), it analyzes the amount of resources required under the predicted load (e.g., initially estimates the need for 2 type-A GPUs). The system determines that the current pool-A has sufficient type-A GPU resources to meet the initial requirements. Based on this, the model adaptation decision is made, and the developer registers model-A on the system management interface and configures the adaptation information: selects compatible manufacturer manufacturer-A and card type type-A, and specifies the deployment method as "containerized deployment". The system confirms that model-A can run on the resources of pool-A based on these information and the current resource pool state. If the predicted load is high or the resources are tight, the system or the developer may choose to optimize model-A, such as selecting a version that supports quantization (e.g., FP16) to reduce GPU memory occupancy, but this step is optional and depends on the initial configuration and prediction results.

[0079] Then, resource dynamic adjustment is performed. Since the predicted load is within an acceptable range and the existing resources are sufficient, this stage may not immediately trigger expansion. However, if the prediction shows that the future load will far exceed the current resource capacity, the system will request more type-A GPU resources from pool-A at this time (for example, by calling the cloud service provider API). In this example, it is assumed that the resources are ready. Therefore, a scheduling decision is made, and the scheduling decision module considers the predicted load (medium), the state of resource pool pool-A (available type-A GPU), the adaptation information of model-A (type-A GPU required), and the preset strategy (such as preferentially using existing resources). The decision result is to deploy the "intelligent customer service assistant" agent to a free type-A GPU node in pool-A, using the registered model-A and its configuration. Thus, the agent instance creation and deployment is performed, and the agent instance management module deploys model-A on the selected node in pool-A through the container image according to the decision, and starts the "intelligent customer service assistant" agent instance instance-A. The system applies the configuration of model-A (such as batch size, quantization settings, etc.). After successful deployment, the instance begins to prepare to receive requests.

[0080] Finally, continuous monitoring and circulation are performed. The system continuously monitors the state (running, health status) and resource usage (GPU utilization, memory, API call rate, response time, etc.) of the newly created instance-A. If the resource usage of instance-A is continuously high and close to the threshold, the system will return to the monitoring and index collection, load prediction and resource demand analysis steps, and re-perform monitoring, prediction, and possibly expansion in the resource dynamic adjustment step (for example, allocate another type-A GPU to the new instance in pool-A), or schedule new requests to other possibly newly created instances in the scheduling decision step. At the same time, the continuous monitoring and circulation step also includes the functions of the monitoring and prediction module, that is, receiving API Key monitoring requests, querying call records, and calculating indicators, such as the total number of calls and the number of failures of the API Key corresponding to instance-A in the past hour, for performance evaluation and further optimization decision. Thus, the deployment of a new agent based on a specific large model is completed.

[0081] In addition, suppose an agent named "intelligent writing assistant" needs to be deployed, which uses a specific LLM model (version v2.1), and the model is recommended to run on an NVIDIA A100 GPU with 24GB of video memory or more for optimal results. The corresponding deployment process is described as follows:

[0082] First, the agent instance management module receives a deployment request. The model adaptation module queries the model metadata and learns that LLM v2.1 requires A100 24GB or above. The resource pool management module reports that there are currently 2 A100 40GB servers (nodes A and B) and 1 V100 16GB server (node C). The model adaptation module recommends using A100 nodes in priority according to hardware affinity. The resource pool management module allocates 1 A100 40GB from node A to the agent instance according to the current load and prediction (assuming that the future traffic will increase). The model adaptation module adjusts the batch size of the model from the default 8 to 12 according to the actual allocated 40GB of video memory, in order to improve throughput.

[0083] Then, the agent instance management module pulls up the agent instance on node A, loads the LLM v2.1 model, and applies the configuration of batch size = 12. After the instance is started, it is registered to the management module and starts processing requests.

[0084] Next, the monitoring and prediction module continuously collects the resource usage of node A (such as GPU utilization stable at 60%), instance performance (average response time 200ms), and API call logs (such as AppID = app-123, API Key = key-123, call volume / hour = 5000).

[0085] Finally, the scheduling decision module analyzes the monitoring data and combines the prediction results (predicting that the traffic will increase by 50% in the next 2 hours) to determine that the current resources may be insufficient. The decision module requests the resource pool management module to allocate 1 A100 40GB on node B. At the same time, it sends instructions to the agent instance management module to start a second instance of the agent (also after model adaptation and resource allocation), and instructs the load balancing to distribute new requests to the two instances. If the traffic is predicted to decrease, or if a performance bottleneck occurs in an instance, the decision module may also trigger the reduction or replacement of the instance.

[0086] Correspondingly, as shown in Figure 5 The embodiments of the present application also provide a large model agent deployment device, which comprises:

[0087] A data determination module 11 is configured to determine agent configuration information of a large model agent to be deployed after receiving an agent deployment request, and determine a corresponding target large model and corresponding model metadata according to the agent configuration information. The agent configuration information includes a preset scheduling policy configured.

[0088] The resource pool query module 12 is configured to determine an available resource condition of a pre-designed computing resource pool, and perform query matching on the pre-designed computing resource pool according to the available resource condition and the model metadata, to determine a plurality of hardware-compatible candidate computing resource nodes.

[0089] The load prediction module 13 is configured to perform a load prediction operation on the to-be-deployed large model agent after being online, to obtain a corresponding predicted load, and generate a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource condition, and the candidate computing resource nodes. The target deployment decision scheme includes a target computing resource node determined from the candidate computing resource nodes.

[0090] The large model loading module 14 is configured to adjust a runtime parameter of the target large model according to a resource characteristic of the target computing resource node, to obtain an adjusted target large model, and load the adjusted target large model on the target computing resource node according to the target deployment decision scheme, to deploy and start an agent instance of the to-be-deployed large model agent.

[0091] In some embodiments, the large model agent deployment apparatus can further include:

[0092] The agent deployment unit is configured to automatically discover a plurality of heterogeneous computing resources available at present, and deploy a resource collection agent to a plurality of computing resource nodes corresponding to the plurality of heterogeneous computing resources, to collect resource information of the plurality of computing resource nodes respectively by the resource collection agent.

[0093] The feature division unit is configured to abstract the resource information into corresponding computing resource labels, and divide according to features of the resource information, to generate the pre-designed computing resource pool for managing the plurality of computing resource nodes.

[0094] In some embodiments, the load prediction module 13 can specifically include:

[0095] The data acquisition unit is configured to acquire historical monitoring data of the to-be-deployed large model agent.

[0096] The trend prediction unit is configured to predict a load change trend in a preset time period after the to-be-deployed large model agent is online, based on the historical monitoring data, by using a time series analysis method or a target machine learning method, to obtain a corresponding predicted load.

[0097] Correspondingly, the load prediction module 13 can specifically include:

[0098] A node determination unit is configured to perform pre-allocation or pre-reservation on the candidate computing resource nodes according to the predicted load, and determine a target computing resource node most suitable for the target large model from the candidate computing resource nodes based on the predicted load, the preset scheduling strategy, and the available resource condition.

[0099] In some embodiments, the load prediction module 13 can specifically include:

[0100] A first decision generation unit is configured to, if the preset scheduling strategy is a first scheduling strategy, generate a corresponding target deployment decision scheme based on the predicted load, the available resource condition, and the candidate computing resource nodes, with the purpose of minimizing cost.

[0101] A second decision generation unit is configured to, if the preset scheduling strategy is a second scheduling strategy, generate a corresponding target deployment decision scheme based on the predicted load, the available resource condition, and the candidate computing resource nodes, with the purpose of optimizing performance.

[0102] A third decision generation unit is configured to, if the preset scheduling strategy is a third scheduling strategy, generate a corresponding target deployment decision scheme based on the predicted load, the available resource condition, and the candidate computing resource nodes, with the purpose of maximizing resource utilization.

[0103] A fourth decision generation unit is configured to, if the preset scheduling strategy is a fourth scheduling strategy, generate a corresponding target deployment decision scheme based on the predicted load, the available resource condition, and the candidate computing resource nodes, with the purpose of minimizing response time.

[0104] In some embodiments, the large model loading module 14 can specifically include:

[0105] A data determination unit is configured to determine model video memory requirement data in the model metadata, and determine available video memory data of the target computing resource node.

[0106] A formula calculation unit is configured to calculate a preset video memory occupancy formula by using the model video memory requirement data and the available video memory data, so as to quantify hardware compatibility between the target large model and the target computing resource node, and obtain a corresponding quantification result.

[0107] A size adjustment unit is configured to adjust a batch processing size of the target large model according to the quantification result.

[0108] In some embodiments, the large model intelligent agent deployment apparatus can further include:

[0109] An instance checking submodule is configured to continuously monitor the agent instance to collect running state information of the agent instance in real time, and perform health checking on the agent instance based on the running state information to determine whether the agent instance is running normally and obtain a corresponding determination result.

[0110] An instance replacing unit is configured to perform an alarm operation on the agent instance and restart or replace the agent instance if the determination result indicates that the agent instance is not running normally.

[0111] Correspondingly, the instance checking submodule can specifically include:

[0112] An index comparison unit is configured to compare a performance index in the running state information with a preset performance index threshold to obtain a corresponding comparison result.

[0113] A result obtaining unit is configured to determine that the performance index is abnormal and obtain the determination result that the agent instance is not running normally if the comparison result indicates that the performance index is higher than the preset performance index threshold.

[0114] In some specific embodiments, the deployment apparatus of the large model agent can further include:

[0115] An instance monitoring unit is configured to continuously monitor the agent instance to obtain a corresponding monitoring result.

[0116] A configuration updating unit is configured to adjust resource allocation of the agent instance, adjust a scheduling weight of the agent instance, perform horizontal scaling operation on the agent instance, or update model configuration of the target large model according to the monitoring result.

[0117] Further, the embodiment of the present application also discloses an electronic device, Figure 6 is an electronic device 20 structure diagram shown according to an exemplary embodiment, the contents in the figure cannot be considered as any limitation on the use range of the present application. The electronic device 20, specifically can include: at least one processor 21, at least one memory 22, power supply 23, communication interface 24, input output interface 25 and communication bus 26. Wherein, the memory 22 is used for storing computer program, the computer program is loaded and executed by the processor 21, to realize the related steps in the large model agent deployment method disclosed in any preceding embodiment. In addition, the electronic device 20 in the embodiment specifically can be electronic computer.

[0118] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0119] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0120] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the deployment method of the large model agent executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0121] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the deployment method of the large model agent disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0122] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between each embodiment, please refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part is described in the method part.

[0123] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0124] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The

[0125] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply or require any such actual relationship or order. Moreover, the terms "include", "contain", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0126] The above provides a detailed description of the technical solutions of the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for deploying large-scale intelligent agents, characterized in that, include: Upon receiving an agent deployment request, the agent configuration information of the large model agent to be deployed is determined, and the corresponding target large model and corresponding model metadata are determined based on the agent configuration information. The agent configuration information includes the configured preset scheduling strategy. The model metadata includes the minimum hardware specifications required for each model version, the recommended hardware specifications, the model file size, and the estimated memory usage during inference. The available resources of the preset computing resource pool are determined, and the preset computing resource pool is queried and matched according to the available resources and the model metadata to determine several hardware-compatible candidate computing resource nodes; A load prediction operation is performed on the large model intelligence agent to be deployed after it goes online to obtain the corresponding predicted load, and a target deployment decision scheme is generated based on the predicted load, the preset scheduling strategy, the available resource status, and the candidate computing resource nodes. The target deployment decision scheme includes the target computing resource node determined from the candidate computing resource nodes; The runtime parameters of the target large model are adjusted according to the resource characteristics of the target computing resource node to obtain the adjusted target large model. The adjusted target large model is then loaded on the target computing resource node according to the target deployment decision scheme to deploy and start the agent instance of the agent to be deployed. The step of adjusting the runtime parameters of the target large model based on the resource characteristics of the target computing resource node includes: Determine the model memory requirement data in the model metadata, and determine the available memory data of the target computing resource node; The preset memory usage formula is calculated using the model's memory requirement data and the available memory data to quantify the hardware compatibility of the target large model and the target computing resource node, and to obtain the corresponding quantification results. The batch size of the target large model is adjusted based on the quantization results.

2. The deployment method for large-scale intelligent agents according to claim 1, characterized in that, Before determining the agent configuration information of the large model agent to be deployed after receiving the agent deployment request, the process also includes: Automatically discover several currently available heterogeneous computing resources and deploy resource collection agents to several computing resource nodes corresponding to the heterogeneous computing resources, so as to collect the resource information of each of the several computing resource nodes through the resource collection agents; The resource information is abstracted into corresponding computing resource tags, and divided according to the characteristics of the resource information to generate the preset computing resource pool for managing the plurality of computing resource nodes.

3. The deployment method for large-scale intelligent agents according to claim 1, characterized in that, The post-deployment load prediction operation for the large model agent to be deployed, to obtain the corresponding predicted load, includes: Obtain historical monitoring data of the large-scale intelligent agent to be deployed; Using time series analysis or target machine learning methods, the load change trend of the large model agent to be deployed is predicted within a preset time period after it goes online, based on the historical monitoring data, so as to obtain the corresponding predicted load. Accordingly, the process of generating a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource status, and the candidate computing resource nodes includes: Based on the predicted load, the candidate computing resource nodes are pre-allocated or reserved. Based on the predicted load, the preset scheduling strategy, and the available resource status, the target computing resource node that best matches the target large model is determined from the candidate computing resource nodes.

4. The deployment method for large-scale intelligent agents according to claim 1, characterized in that, The process of generating a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource status, and the candidate computing resource nodes includes: If the preset scheduling strategy is the first scheduling strategy, then with the aim of minimizing costs, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource status, and the candidate computing resource nodes. If the preset scheduling strategy is the second scheduling strategy, then with the goal of optimal performance, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource status, and the candidate computing resource nodes. If the preset scheduling strategy is the third scheduling strategy, then with the goal of maximizing resource utilization, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource status, and the candidate computing resource nodes. If the preset scheduling strategy is the fourth scheduling strategy, then with the aim of minimizing response time, a corresponding target deployment decision scheme is generated based on the predicted load, the available resource status, and the candidate computing resource nodes.

5. The deployment method for large-scale intelligent agents according to claim 1, characterized in that, After deploying and starting the agent instance of the large model agent to be deployed, the process further includes: The agent instance is continuously monitored to collect its running status information in real time, and a health check is performed on the agent instance based on the running status information to determine whether the agent instance is running normally and to obtain the corresponding judgment result. If the judgment result indicates that the agent instance is not running normally, then an alarm operation is performed on the agent instance, and the agent instance is restarted or replaced. Accordingly, the step of performing a health check on the agent instance based on the running status information to determine whether the agent instance is running normally and obtaining the corresponding judgment result includes: The performance indicators in the running status information are compared with preset performance indicator thresholds to obtain the corresponding comparison results; If the comparison result indicates that the performance index is higher than the preset performance index threshold, then the performance index is determined to be abnormal, and the result of the judgment that the agent instance is not running normally is obtained.

6. The deployment method for large-scale intelligent agents according to claim 1, characterized in that, After deploying and starting the agent instance of the large model agent to be deployed, the process further includes: Continuously monitor the agent instances to obtain corresponding monitoring results; Adjust the resource allocation of the agent instance based on the monitoring results, or adjust the scheduling weight of the agent instance, or perform horizontal scaling operations on the agent instance, or update the model configuration of the target large model.

7. A deployment device for a large-scale intelligent agent, characterized in that, include: The data determination module is used to determine the agent configuration information of the large model agent to be deployed after receiving the agent deployment request, and to determine the corresponding target large model and corresponding model metadata based on the agent configuration information; the agent configuration information includes the configured preset scheduling strategy; the model metadata includes the minimum hardware specifications required for each model version, the recommended hardware specifications, the model file size, and the estimated memory usage during inference; The resource pool query module is used to determine the available resources of a preset computing resource pool, and to query and match the preset computing resource pool based on the available resources and the model metadata to determine several hardware-compatible candidate computing resource nodes. The load prediction module is used to perform load prediction operations on the large model intelligence agent to be deployed after it goes online, so as to obtain the corresponding predicted load, and generate a target deployment decision scheme based on the predicted load, the preset scheduling strategy, the available resource status, and the candidate computing resource nodes. The target deployment decision scheme includes the target computing resource node determined from the candidate computing resource nodes; The large model loading module is used to adjust the runtime parameters of the target large model according to the resource characteristics of the target computing resource node to obtain the adjusted target large model, and load the adjusted target large model on the target computing resource node according to the target deployment decision scheme to deploy and start the agent instance of the agent to be deployed large model agent; The large model loading module includes: The data determination unit is used to determine the model memory requirement data in the model metadata and to determine the available memory data of the target computing resource node. The formula calculation unit is used to calculate a preset memory usage formula using the model memory requirement data and the available memory data, so as to quantify the hardware compatibility of the target large model and the target computing resource node and obtain the corresponding quantification results. The size adjustment unit is used to adjust the batch size of the target large model according to the quantization result.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the deployment method of a large model agent as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the deployment method of a large model agent as described in any one of claims 1 to 6.

Images