Method, device and equipment for large model inference optimization and storage medium

Abstract

The application discloses a large model inference optimization method and device, equipment and a storage medium, and belongs to the technical field of computers. The method specifically comprises the following steps: acquiring hardware perception information and inference task information of a large model inference system, wherein the hardware perception information comprises static environment information and dynamic running information; based on a preset event triggering condition, the inference task information and the hardware perception information, utilizing a hardware evaluation strategy to determine to-be-optimized event information; based on a preset knowledge base and the to-be-optimized event information, utilizing an optimization algorithm to obtain an optimized arrangement scheme; and based on the optimized arrangement scheme, performing optimization processing on the large model inference system.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the fields of big data analysis technology and large model inference technology, and particularly to a method, apparatus, device and storage medium for large model inference optimization. Background Technology

[0002] Distributed large-scale model inference involves a huge number of model parameters, and during inference, intermediate activation values ​​and key-value caches need to be stored, consuming a large amount of GPU memory. Without optimization, a model often cannot complete inference.

[0003] Currently, while large-model distributed inference frameworks in related technologies have made improvements in model parallelism, communication optimization, memory management, and batch scheduling, they still lack a globally adaptive workflow orchestration mechanism overall. Many solutions require manual pre-configuration of parameters, such as parallelism and sharding strategies, which are not easily changed during operation. They cannot automatically adapt when the environment or load changes, which may lead to idle resources or local overload in some cases, thereby reducing throughput or increasing latency. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for optimizing large-scale model inference, which can solve problems such as high communication costs between different devices, uneven load distribution, and high system optimization complexity in practical large-scale model distributed inference scenarios. The technical solution is as follows: Firstly, a method for optimizing large model inference is provided, the method comprising: Acquire hardware awareness information and inference task information of a large model inference system, wherein the hardware awareness information includes static environment information and dynamic operation information; Based on preset event triggering conditions, the inference task information, and the hardware perception information, the event information to be optimized is determined using a hardware evaluation strategy. Based on the preset knowledge base and the event information to be optimized, an optimized orchestration scheme is obtained using an optimization algorithm; Based on the optimized orchestration scheme, the large model inference system is optimized.

[0005] In one possible implementation, acquiring the hardware-aware information and inference task information of the large model inference system includes: In response to the startup of the inference service of the large model inference system, the static environment information is obtained; In response to the large model inference system performing inference services based on the received inference request and initial orchestration scheme, the dynamic operation information is obtained; Based on the inference request received by the large model inference system, inference task information is obtained.

[0006] In one possible implementation, the step of determining the event information to be optimized based on preset event triggering conditions, the inference task information, and the hardware-aware information, using a hardware evaluation strategy, includes: Based on the preset event triggering conditions and the hardware sensing information, determine the event information that satisfies the preset event triggering conditions; Based on the inference task information, a hardware evaluation strategy is used to evaluate and process event information that meets the preset event triggering conditions. Based on the results of the evaluation process, the event information to be optimized is determined.

[0007] In one possible implementation, determining the event information to be optimized based on the result of the evaluation process includes: The results of the evaluation process are then denoised. Based on the results of the denoising process, the event information to be optimized is determined.

[0008] In one possible implementation, the step of obtaining an optimized orchestration scheme based on the preset knowledge base and the event information to be optimized using an optimization algorithm includes: Based on the preset knowledge base, the constraints and heuristic information of the optimization algorithm are determined to construct the optimization algorithm; Based on the event information to be optimized, the expected benefits of the candidate orchestration schemes corresponding to the event information to be optimized are obtained using the optimization algorithm. Based on the expected benefits of the candidate orchestration schemes, the candidate orchestration schemes are evaluated to obtain the optimized orchestration scheme.

[0009] In one possible implementation, optimizing the large-model inference system based on the optimized orchestration scheme includes: Based on the optimized orchestration scheme, the current orchestration scheme of the large model inference system is adjusted. Obtain the adjusted hardware perception information of the large model inference system; The adjusted hardware perception information is evaluated and processed. If the result of the evaluation process does not meet the preset evaluation conditions, the orchestration scheme of the large model inference system is reverted to the current orchestration scheme before the adjustment process.

[0010] Secondly, an apparatus for optimizing large model inference is provided, the apparatus comprising: The acquisition unit is used to acquire hardware perception information and inference task information of the large model inference system. The hardware perception information includes static environment information and dynamic operation information. The determining unit is used to determine the event information to be optimized based on preset event triggering conditions, the inference task information, and the hardware perception information, using a hardware evaluation strategy. The obtaining unit is used to obtain an optimized orchestration scheme based on the preset knowledge base and the event information to be optimized, using an optimization algorithm. The optimization unit is used to optimize the large model inference system based on the optimized orchestration scheme.

[0011] Thirdly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects and any possible implementations described above.

[0012] Fourthly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0013] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0014] The beneficial effects of the technical solution provided in this application include at least the following: As can be seen from the above technical solution, the embodiments of this application can obtain hardware awareness information and inference task information of a large model inference system. The hardware awareness information includes static environment information and dynamic operation information. Then, based on preset event triggering conditions, the inference task information, and the hardware awareness information, a hardware evaluation strategy can be used to determine the event information to be optimized. Based on the preset knowledge base and the event information to be optimized, an optimization algorithm can be used to obtain an optimized orchestration scheme. Based on the optimized orchestration scheme, the large model inference system can be optimized. Since the hardware usage of the obtained hardware awareness information and inference task information can be evaluated, the event information that needs to be optimized can be dynamically determined. Then, the optimization algorithm can be used to obtain the optimized orchestration scheme corresponding to the event information that needs to be optimized. The optimized orchestration scheme can be used to effectively optimize the current orchestration scheme of the large model inference system, realizing adaptive orchestration optimization of the workflow in distributed large model inference. The orchestration scheme of the inference workflow is dynamically adjusted, breaking through the limitations of traditional static configuration. The resource configuration is dynamically optimized according to the operating status, which can avoid unit bottlenecks dragging down the whole, improve the average throughput, optimize latency, and improve the resource utilization of the system during operation, thereby ensuring the performance and stability of the distributed large model inference system.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for optimizing large model inference according to an embodiment of this application; Figure 2 This is a flowchart illustrating a method for optimizing large model inference according to another embodiment of this application; Figure 3 This is a schematic diagram illustrating an application scenario of the architecture of the large model inference optimization method provided in another embodiment of this application; Figure 4 This is a structural block diagram of a large model inference optimization apparatus provided in another embodiment of this application; Figure 5 This is a block diagram of an electronic device used to implement the large model inference optimization method of the embodiments of this application. Detailed Implementation

[0018] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0021] Currently, while large-model distributed inference frameworks have made improvements in model parallelism, communication optimization, memory management, and batch scheduling, they still lack a globally adaptive workflow orchestration mechanism. Many solutions require manual pre-configuration of parameters (such as parallelism and sharding strategies) that are difficult to change during operation and cannot automatically adapt to changes in environment or load. This can lead to idle resources or local overload in some cases, thereby reducing throughput or increasing latency. For example, DeepSpeed ​​requires users to select the parallelism, and static configuration may not be optimal if hardware is updated or request patterns change. ORCA uses iterative scheduling to improve GPU utilization, but it assumes that model sharding and resource configuration are fixed and does not address how to adjust based on different hardware capabilities. TensorRT-LLM mainly optimizes memory and computing power on a single machine and does not involve comprehensive strategies for resource orchestration in distributed scenarios. Even NVIDIA Dynamo, which introduces runtime planners and intelligent routers for dynamic optimization in specific scenarios (pre-filling / decoding separation, KV cache reuse), has its tuning scope limited to a few preset modes and is not yet a general intelligent orchestration agent. Due to the lack of a unified adaptive scheduling mechanism, existing systems struggle to adjust in a timely manner to changing hardware environments and load demands, leading to performance bottlenecks and resource waste. For example, solutions lacking hardware awareness cannot optimize communication and computing strategies based on the presence or absence of RDMA or differences in GPU models, resulting in high communication costs on low-end networks or underutilization of bandwidth resources on high-end networks. Furthermore, systems without event-triggered adjustment may experience response latency exceeding SLO requirements when faced with sudden surges in long text requests due to a failure to allocate resources in a timely manner.

[0022] Therefore, there is an urgent need for a method to optimize large-scale model inference, which can effectively perceive environmental changes and optimize resource allocation and model partitioning in real time, thereby ensuring adaptability and stability in distributed large-scale model inference.

[0023] Please refer to Figure 1 This document illustrates a flowchart of a large model inference optimization method provided in one embodiment of this application. This large model inference optimization method can be applied to intelligent agents and may specifically include: Step 101: Obtain hardware awareness information and inference task information of the large model inference system. The hardware awareness information includes static environment information and dynamic operation information.

[0024] Step 102: Based on the preset event triggering conditions, the inference task information, and the hardware perception information, determine the event information to be optimized using a hardware evaluation strategy.

[0025] Step 103: Based on the preset knowledge base and the event information to be optimized, an optimized orchestration scheme is obtained using an optimization algorithm.

[0026] Step 104: Based on the optimized orchestration scheme, optimize the large model inference system.

[0027] It should be noted that hardware-aware information may include, but is not limited to, a list of GPU nodes, the number of GPUs on each node, GPU models, GPU memory capacity, system CPU performance, system CPU memory size, network topology information, and available software acceleration libraries. Network topology information may include bandwidth, latency, and whether RDMA is supported. Available software acceleration libraries may include cuBLAS LT, TensorRT, and MPI library versions.

[0028] It should be noted that dynamic operational information may include, but is not limited to, real-time memory usage and utilization of each GPU, inter-GPU communication bandwidth utilization, and application-level metrics. Application-level metrics may include current request queue length, response latency, processing time for each request, token generation rate, and key-value cache hit rate. Here, dynamic operational information may include dynamic performance information of the system during the inference service process.

[0029] It should be noted that the optimized orchestration scheme can be the orchestration scheme corresponding to the event information to be optimized. This optimized orchestration scheme can be pre-configured according to actual business needs. The orchestration scheme can include quantization schemes, distributed schemes, communication schemes, and prefill-decode (PD) schemes.

[0030] It should be noted that the inference task information may include the task probability distribution and the task type. The task probability distribution may include the input length distribution and the output length distribution. The task type may include, but is not limited to, stories, code, chat, data processing, and scheduled tasks.

[0031] In this way, by evaluating hardware usage based on the acquired hardware awareness information and inference task information, the event information that needs optimization can be dynamically determined. Then, by using optimization algorithms, the corresponding optimization orchestration scheme can be obtained. The current orchestration scheme of the large model inference system can be effectively optimized using the optimization orchestration scheme, realizing adaptive orchestration optimization of workflow in distributed large model inference. It dynamically adjusts the orchestration scheme of the inference workflow, breaks through the limitations of traditional static configuration, and dynamically optimizes resource allocation according to the running status. This avoids unit bottlenecks dragging down the whole, improves average throughput, optimizes latency, and improves the resource utilization of the system during operation, thereby ensuring the performance and stability of the distributed large model inference system.

[0032] Optionally, in one possible implementation of this embodiment, in step 101, firstly, in response to the inference service startup of the large model inference system, the static environment information is obtained. Secondly, in response to the large model inference system performing inference service based on the received inference request and initial orchestration scheme, the dynamic operation information can be obtained. Thirdly, inference task information can be obtained based on the inference request received by the large model inference system.

[0033] In this implementation, the inference request may include question information sent by the user to the large model inference system. The initial orchestration scheme may be the currently preset orchestration scheme of the large model inference system.

[0034] In a specific implementation of this approach, when the inference service of the large model inference system starts, the static environment information can be obtained; when the large model inference system executes the inference service based on the received inference request and the initial orchestration scheme, dynamic operation information is obtained in real time.

[0035] In another specific implementation of this approach, based on the inference request received by the large model inference system, inference task information is obtained from the user-side agent. The inference task information may include the task probability distribution and the task type.

[0036] In this way, static environmental information and dynamic operational information can be dynamically and in real time perceived during the inference service of the large model inference system, so as to identify more accurate and effective events to be optimized and to accurately and timely determine whether the orchestration scheme of the large model inference system needs to be adjusted.

[0037] Optionally, in one possible implementation of this embodiment, in step 102, firstly, event information that satisfies the preset event triggering conditions can be determined based on the preset event triggering conditions and the hardware awareness information. Secondly, the event information that satisfies the preset event triggering conditions can be evaluated using a hardware evaluation strategy based on the inference task information. Thirdly, event information to be optimized can be determined based on the result of the evaluation process.

[0038] In this implementation, the preset event triggering conditions may include the hardware sensing information exceeding a preset security threshold, or the hardware sensing information showing a significant trend of change.

[0039] In one specific implementation of this approach, the inference task information may include a task probability distribution and a task type. Then, based on the task probability distribution and task type, a hardware evaluation strategy can be used to evaluate and process event information that meets the preset event triggering conditions. Furthermore, based on the results of the evaluation and processing, event information to be optimized can be determined.

[0040] One specific implementation involves determining the hardware resource requirements corresponding to the task probability distribution and task type. Based on these requirements, it is then determined whether event information meeting the preset event triggering conditions can handle the necessary hardware resources. If the event information cannot meet the hardware resource requirements, it is identified as an event to be optimized. If the event information can meet the hardware resource requirements, no processing is performed.

[0041] For example, if the event information is that the average request latency exceeds 100ms within the last second, and the task type is a short question-and-answer chat, then the event information can be processed. If the task type is telling a longer fairy tale, then the event information can be identified as an event information to be optimized.

[0042] In another specific implementation of this approach, firstly, the result of the evaluation process can be denoised. Then, based on the result of the denoising process, the event information to be optimized is determined.

[0043] In this specific implementation process, the evaluation results may include information on candidate events to be optimized.

[0044] One specific implementation method is to determine the event information to be optimized by associating and merging candidate event information based on the type of event information to be optimized.

[0045] Another specific implementation process is to determine the queue order of the event information to be optimized based on the priority level of the event information to be optimized, and then use the optimization algorithm and the preset knowledge base to obtain the optimization arrangement scheme corresponding to the event information to be optimized.

[0046] For example, the priority level can be determined by actual business needs. For instance, events that affect user latency have a higher priority than background resource optimization events.

[0047] In this way, by using hardware evaluation strategies based on inference task information, event information that meets preset event triggering conditions can be evaluated and processed to determine accurate and effective event information to be optimized, so that optimization strategies can be dynamically switched for different load modes through event triggering.

[0048] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the aforementioned implementation methods to achieve the large model inference optimization method of this embodiment. Detailed descriptions can be found in the relevant content of the aforementioned implementation methods, and will not be repeated here.

[0049] Optionally, in one possible implementation of this embodiment, in step 103, firstly, the constraints and heuristic information of the optimization algorithm can be determined based on the preset knowledge base to construct the optimization algorithm. Secondly, based on the event information to be optimized, the optimization algorithm can be used to obtain the expected benefits of the candidate orchestration schemes corresponding to the event information to be optimized. Thirdly, based on the expected benefits of the candidate orchestration schemes, the candidate orchestration schemes can be evaluated to obtain the optimized orchestration scheme.

[0050] In this implementation, the optimization algorithm may include optimization algorithms based on heuristic search, optimization algorithms based on reinforcement learning strategies, etc.

[0051] In one specific implementation of this approach, based on the preset knowledge base, the constraints and heuristic information of the optimization algorithm are determined to construct a heuristic search-based optimization algorithm. Secondly, based on the event information to be optimized, the expected returns of candidate orchestration schemes corresponding to the event information can be obtained using the heuristic search-based optimization algorithm. Thirdly, based on the expected returns of the candidate orchestration schemes, the candidate orchestration schemes can be evaluated to obtain an optimized orchestration scheme that meets the preset performance targets and constraints.

[0052] In this implementation, the pre-set knowledge base may include pre-set expert rules, optimization measures rules for specific scenarios, etc.

[0053] In another specific implementation of this approach, the rules of a preset knowledge base can be queried based on the event information to be optimized to obtain the optimization orchestration scheme corresponding to the event information to be optimized.

[0054] In this way, by utilizing a pre-set knowledge base, the constraints and heuristic information of the optimization algorithm can be determined to construct the optimization algorithm. Using the optimization algorithm, the optimization orchestration scheme corresponding to the event information to be optimized can be obtained, which can further improve the reliability and accuracy of the optimization orchestration scheme.

[0055] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the aforementioned implementation methods to achieve the large model inference optimization method of this embodiment. Detailed descriptions can be found in the relevant content of the aforementioned implementation methods, and will not be repeated here.

[0056] Optionally, in one possible implementation of this embodiment, in step 104, firstly, the current orchestration scheme of the large model inference system can be adjusted based on the optimized orchestration scheme. Secondly, the adjusted hardware awareness information of the large model inference system is obtained. Thirdly, the adjusted hardware awareness information is evaluated. Fourthly, in response to the evaluation result not meeting the preset evaluation conditions, the orchestration scheme of the large model inference system is reverted to the current orchestration scheme before the adjustment.

[0057] In this implementation, the preset evaluation conditions may include the absence of corresponding event information to be optimized in the hardware-aware information of the adjusted large model inference system.

[0058] In one specific implementation of this approach, based on preset event triggering conditions, the inference task information, and the hardware awareness information, a hardware evaluation strategy is used to determine whether there is any event information to be optimized. If not, the large model inference system can be controlled to provide inference services according to the optimized orchestration scheme. If so, the orchestration scheme of the large model inference system can be reverted to the current orchestration scheme before the adjustment process, and the large model inference system can be controlled to provide inference services according to the current orchestration scheme.

[0059] Understandably, if the large model inference system after the orchestration scheme is not equipped with the previous event information, it can continue to provide services according to the optimized orchestration scheme. If the previous problem is not resolved, it can revert to the previous orchestration scheme.

[0060] In this way, the results of adjusting the orchestration scheme can be monitored to evaluate the effect of the adjustment. If the adjustment is found to be unsatisfactory or to have a negative impact, such as increased latency, the rollback logic can be triggered to restore the system to the orchestration state before the adjustment. This can avoid the instability that may be caused by frequent adjustments, and ensure the smooth transition of the system and the robustness of system optimization.

[0061] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the aforementioned implementation methods to achieve the large model inference optimization method of this embodiment. Detailed descriptions can be found in the relevant content of the aforementioned implementation methods, and will not be repeated here.

[0062] To better understand the methods of the embodiments of this application, the methods of the embodiments of this application will be described below in conjunction with the accompanying drawings and specific application scenarios.

[0063] Figure 2 This is a flowchart illustrating a method for optimizing large model inference according to another embodiment of this application, as shown below. Figure 2As shown. In this embodiment, in this application scenario, the large model inference optimization method can be applied to an intelligent agent, specifically including: Step 201: Obtain hardware awareness information and inference task information of the large model inference system.

[0064] In this embodiment, the hardware-aware information includes static environmental information and dynamic operational information.

[0065] In this embodiment, for example, when the large model inference service starts, the agent first performs hardware environment probing. It can scan all participating nodes to obtain the GPU model, memory capacity, and computing power (FLOPS) of each server. It can also confirm the connection type and speed between nodes through network probing. For example, whether they are under the same high-speed interconnect switch, whether RDMA passthrough is supported, etc. It can also load software environment information, such as GPU driver version, availability and version number of library dependencies, such as cuBLAS, NCCL, TensorRT, etc. Secondly, based on static environment information, it determines a recommended configuration that matches the current environment, i.e., the initial orchestration scheme. For example, if it detects that each GPU has sufficient memory and high bandwidth, it tends to use cross-device tensor parallelism to reduce single-GPU memory usage; if it finds that some GPUs have limited memory or slow networks, it tends to use pipelined parallelism to place model chunks on GPUs of the same node to reduce communication overhead. The initial orchestration scheme may include model partitioning mapping, parallel mode selection, and scheduling strategies. For example, model partitioning mapping may include the GPU where each layer or module resides. Parallelism mode selection can include the number of data parallel instances, tensor parallelism, and whether pre-decoupling is required. Scheduling strategies can include batch size and routing rules. Next, model replicas and shards are initialized according to the initial orchestration scheme, model weights are loaded onto each GPU, and necessary communication channels are established. At this point, the large model inference system enters a standby state.

[0066] When a user's inference request arrives, the large model inference system begins processing it in parallel according to a predetermined initial orchestration scheme. Taking a typical LLM text generation request as an example: the input information is first processed by a GPU set in the pre-filling stage. If no separation is used, this can be done on a single GPU; or, if PD separation is used, the pre-filling cluster can process it on a group of GPUs. The pre-filling computation yields an initial output draft (token) and a context key-value cache. Then, the system enters a cyclical decoding stage, where the next token is generated by the decoding stage GPU set at each step, until the required length is reached. During the inference service, on the one hand, if the model is split across multiple GPUs, the synchronous or pipelined operation of each part can be coordinated. For example, layer-by-layer pipelines can pass the output of the first layer to the GPU containing the second layer, and so on, or partial results from each GPU computation can be collected in tensor parallelism and aggregated before being sent to the next layer. On the other hand, data exchange between GPUs is performed according to optimized paths, such as RDMA or NVLink, and overlaps with computation to hide latency. For example, while the GPU computes the next layer, the key-value cache of the previous layer is asynchronously transmitted via the NIXL library. For multiple concurrent requests, they can be assigned to GPUs utilizing the same key-value cache based on cache similarity to avoid redundant computation. The entire inference task service can run stably under the initial orchestration scheme set by the agent. During system operation, the agent can continuously collect various metrics, i.e., dynamic operational information.

[0067] Step 202: Based on the preset event triggering conditions and hardware perception information, determine the event information that meets the preset event triggering conditions.

[0068] Step 203: Based on the inference task information, use the hardware evaluation strategy to evaluate and process the event information that meets the preset event triggering conditions.

[0069] Step 204: Denoise the evaluation results to determine the event information to be optimized.

[0070] In this embodiment, it can be determined whether the hardware sensing information meets the preset event triggering conditions. If it does, the event information that meets the preset event triggering conditions can be identified.

[0071] For example, if the acquired hardware-aware information indicates a large influx of requests within a short period with a long average input sequence, GPU computation queue backlog during the pre-filling phase, and a rapid increase in processing latency, it can be determined whether the pre-filling GPU utilization is consistently at 100% and whether the queue length exceeds a preset safety threshold. In other words, it can be determined whether the hardware-aware information meets the preset event triggering conditions. If it does, the candidate event information can be identified as insufficient pre-filling computing power. If the acquired hardware-aware information indicates that GPU memory is nearly exhausted, triggering the framework's memory paging mechanism and causing latency jitter, it can be determined whether GPU memory usage exceeds 90% and data overflows to the CPU. If so, the candidate event information can be identified as a memory pressure event. If the acquired hardware-aware information indicates network congestion or RDMA link failure, increasing cross-node communication latency, it can be determined whether the average communication latency exceeds a preset latency threshold, for example, X times higher than the usual latency. If so, the candidate event information can be identified as a communication pressure event. Furthermore, based on inference task information, the candidate event information can be evaluated and processed, and the evaluated information can be denoised and sorted to obtain the final event information to be optimized.

[0072] Step 205: Utilize a pre-set knowledge base and heuristic search algorithm to optimize the event information to obtain an optimized orchestration scheme.

[0073] Step 206: Adjust the current orchestration scheme of the large model inference system to an optimized orchestration scheme, so as to control the large model inference system to perform inference services based on the optimized orchestration scheme.

[0074] In this embodiment, linear programming or heuristic search can be used to optimize the objective function (throughput / latency) under constraints, or a trained reinforcement learning agent can be run to determine the optimal orchestration scheme. First, based on the information of the event to be optimized, corresponding optimization strategies can be found. By combining current environmental information and historical states, one or a set of executable adjustment schemes can be generated, and their expected benefits and costs can be evaluated. Then, the optimal scheme, i.e., the optimized orchestration scheme, is selected.

[0075] For example, when the event information to be optimized is that the pre-filling GPU is overloaded while the decoding GPU is idle, based on a preset knowledge base, the system retrieves two suggested solutions: Solution 1, "Temporarily allocate decoding GPUs to assist in pre-filling, and convert one decoding stage GPU to be used for pre-filling," and Solution 2, "Switch some batches to be pre-filled by decoding GPUs." After a heuristic search algorithm simulates and evaluates the latency impact, the solution with the highest benefit is selected to obtain the optimized orchestration scheme. Considering that adding decoding GPUs for pre-filling may lead to slightly higher KV transmission latency, but can alleviate the computational bottleneck, the optimized orchestration scheme can be Solution 1. This allows a pre-filling computation thread to be started on the decoding GPU, directly assigning some newly arriving requests to this thread for processing, updating the request routing rules, and causing subsequent long text requests to be hashed and diverted to the previously idle decoding GPUs for pre-filling computation. In this way, the pre-filling load is distributed, the pressure on the previously overloaded GPUs is reduced, and the overall system throughput increases.

[0076] For example, when the event to be optimized is a memory pressure event, the optimization assessment prioritizes ensuring memory stability. The optimization orchestration plan might involve triggering model resharding, transferring some model weights or key-value caches to another GPU with spare memory, or enabling a tiered storage caching scheme to offload older key-value cache entries to CPU memory / local SSD, trading space for time to alleviate memory pressure. Furthermore, infrequently accessed caches can be moved to main memory, and the model sharding management can be notified to offload some intermediate layer calculations for the next batch of inference to a backup GPU.

[0077] For example, when the event information to be optimized is a communication pressure event, communication becomes the bottleneck. Optimizing the orchestration scheme can reduce cross-node communication. Specifically, the current cross-node model parallelism can be changed to local execution, completing a portion of the inference within each node before merging the results, or the tensor parallelism can be temporarily reduced. Simultaneously, fault-tolerant features such as flow control can be enabled to prevent further delays caused by congestion and retries. Subsequent scheduling can be adjusted without affecting the service of inference tasks already in progress.

[0078] In this embodiment, the intelligent agent can further monitor the execution of the adjusted optimized orchestration scheme.

[0079] For example, in scenarios where the optimized orchestration scheme involves temporarily allocating decoding GPUs to assist in pre-filling, it's possible to further monitor whether the pre-filling GPU utilization drops to a reasonable range and whether throughput increases. For scenarios where the optimized orchestration scheme triggers model re-splitting, it's possible to monitor whether memory usage decreases and whether swapping ceases. For scenarios where the optimized orchestration scheme reduces cross-node communication, it's possible to determine whether average communication latency has returned to normal. If the target effect is not achieved or new problems arise, such as causing the decoding GPU to be slightly busy but still have spare capacity, the agent can further fine-tune the scheme, for example, by adding another GPU to participate in pre-filling. Or, if new problems arise, it can first revert to the initial orchestration scheme. If the adjustment is effective, the system will run stably under the new orchestration scheme. Afterwards, if the load characteristics return to normal, for example, long text requests decrease, the agent can also detect the rebalancing of pre-filling and decoding loads and determine whether to revert the aforementioned adjustments to release resources, i.e., whether to trigger a reverse event, such as "pre-filling GPU utilization remains below 50% and decoding GPU is slightly strained," in which case the previously allocated orchestration scheme will be reverted. The entire process is repeated cyclically, enabling continuous self-optimization control of system operation.

[0080] In this embodiment, Figure 3 This is a schematic diagram illustrating an application scenario of the architecture of the large model inference optimization method provided in one embodiment of this application, such as... Figure 3 As shown, in the application scenario, an intelligent agent can include a perception layer, a decision-making layer, an execution layer, and a monitoring layer.

[0081] Preferably, the perception layer here may include a hardware and environment perception module, a runtime monitoring module, and an event bus and analysis module. The hardware and environment perception module is responsible for collecting underlying environmental information during system initialization and operation. This module can obtain information such as the GPU configuration (number, model, memory capacity, etc.) of each computing node, system configuration (CPU performance, memory size, network topology such as bandwidth and latency, whether RDMA is supported, and available software acceleration libraries such as installed cuBLAS LT, TensorRT, and MPI library versions) by calling methods such as the NVIDIA Management Library (NVML), operating system interfaces, and network probing tools. This information is stored in a structured manner as hardware environment information; the hardware perception module acts as the "sensors" of the intelligent agent, enabling it to have a comprehensive understanding of its environment. The runtime monitoring module can continuously collect dynamic performance data, i.e., dynamic runtime information, of the system during inference services. The runtime monitoring module includes monitoring of GPU utilization and memory usage, such as recording the percentage of computational utilization for each GPU and the total memory usage (MB / total) every few milliseconds; monitoring of communication queues and bandwidth, such as statistical analysis of GPU data transfer volume and latency between nodes; and monitoring of application-level metrics, such as the number of currently queued requests, processing time for each request, token generation rate, and KV cache hit rate. This module can also perform event analysis on the raw hardware awareness information. When hardware awareness information exceeds preset thresholds or shows a significant trend, corresponding event notifications are generated. For example, "GPU 0 memory usage above 90%" is marked as a memory pressure event, and "average request latency exceeding 100ms in the last second" triggers performance degradation events and communication pressure events. The event bus and analysis module aggregates the generated event information to the event bus. The event bus is responsible for queuing various events according to priority and provides a publish / subscribe mechanism so that the decision-making layer module can receive the event types of interest. Simultaneously, a simple event analysis unit performs correlation and noise reduction on the events, preventing the decision-making layer from being disturbed by massive amounts of redundant events. For example, multiple consecutive high memory usage events can be merged into a single "sustained memory high pressure" event, and overload events on different GPUs can be comprehensively evaluated to determine if they belong to the same cause. The analyzed events are prioritized and sent sequentially to the adaptive decision engine in the decision layer according to the event queue.

[0082] Preferably, the decision layer here may include an adaptive decision engine. The adaptive decision engine can be considered the brain of the entire agent. This engine has built-in multiple policy algorithms to formulate adjustment decisions, i.e., optimize orchestration schemes, based on the current system state, i.e., the event information sent by the perception layer. The decision engine may include a preset rule base / knowledge base and optimization algorithms. The preset rule base / knowledge base stores optimization rules for specific scenarios, such as the knowledge rule "if RDMA is detected to be unavailable and cross-node communication is high, then reduce model parallelism," and the criterion "if GPU utilization is uneven, then try to reallocate model layers." Optimization algorithms may include linear programming algorithms, heuristic search algorithms, and reinforcement learning algorithms. For example, linear programming or heuristic search can be used to optimize the objective function under constraints, or a trained reinforcement learning agent can be run to determine the optimal orchestration scheme. After receiving an event, the decision engine searches for the corresponding optimization strategy, integrates the current environmental information and historical states, generates one or a set of executable adjustment schemes, and evaluates their expected benefits and costs. Then, the optimal scheme is selected and submitted to the execution layer.

[0083] Preferably, the execution layer here may include a workflow orchestration and execution control module, which can directly affect the configuration and scheduling of the underlying inference service. This module receives information from the decision engine and calls specific control interfaces to adjust the behavior of the large model inference system. The workflow orchestration and execution control module may include a model sharding management submodule, a task scheduling and resource allocation submodule, and a communication and acceleration control submodule. The model sharding management submodule can be responsible for the deployment layout of the model across multiple devices and can dynamically change the model sharding method according to the orchestration scheme. For example, it supports moving certain neural network layers to a specified GPU at the hierarchical granularity, or adjusting the sharding dimension and the number of participating GPUs in tensor parallelism. In terms of implementation, it can use tensor parallelism libraries, such as the tensor parallelism implementation of Megatron-LM, or the distributed checkpoint loading function provided by the framework to remap and load model weights on different devices. This submodule ensures the consistency of each part of the model and maintains existing optimizations during the repartitioning process, such as retaining the compiled TensorRT engine and transferring it to the new device for reuse. The task scheduling and resource allocation submodule controls how inference tasks are scheduled across various GPUs for execution, such as pre-filling or decoding computations for a batch of requests. It corresponds to model sharding layout; if the model is split across multiple GPUs, it needs to coordinate pipeline scheduling; if GPU resources are added or removed at a certain stage, it is responsible for updating the scheduling strategy, such as updating the load balancing algorithm to allocate more requests to newly added GPU workers. This submodule can link with the underlying inference service framework, dynamically adding or removing model instances and replicas, and changing routing strategies through its API interface to achieve resource reallocation. The communication and acceleration control submodule can be used to adjust communication modes and computation kernels according to optimized orchestration schemes. In terms of communication control, for example, when receiving an optimized orchestration scheme that enables RDMA optimization, this submodule can call the underlying library to redirect all cross-node GPU memory transfers to the GPUDirect RDMA channel; another example is an optimized orchestration scheme that reduces communication frequency, which can enable communication compression or gradient accumulation to reduce the number of communication round trips. In terms of computational acceleration, it can be responsible for loading or unloading specific optimization library modules at appropriate times based on the optimized orchestration scheme. For example, it can switch certain operators to TensorRT implementations (the Engine needs to be pre-compiled) or switch between high-precision and low-precision kernels.

[0084] The monitoring layer can include feedback and rollback mechanisms. These mechanisms mean that the execution control module also includes feedback monitoring of the results of adjustment actions, as well as rollback functionality when necessary. After each optimization and orchestration adjustment determined by the agent, this module can briefly monitor key indicators to evaluate the adjustment's effectiveness. If the adjustment fails to meet expectations or even has negative impacts, such as increased latency, rollback logic can be triggered to restore the system to its pre-adjustment configuration state. This ensures the robustness of the solution and avoids instability that frequent adjustments may cause.

[0085] It is understandable that, such as Figure 3 As shown, in the software architecture of this application scenario, the core intelligent agent can reside in the control plane, while the model management, scheduling execution, and communication optimization modules can belong to the data plane or the actual working plane. The two can interact through clearly defined interfaces. This architectural design ensures the decoupling of the agent's decision-making logic from the underlying inference execution, facilitating expansion and maintenance. Simultaneously, the modules can be deployed in a distributed manner: the monitoring agent and communication optimization run locally on each node, the core intelligent agent makes decisions centrally, and model management and scheduling have both centralized and decentralized execution components. Overall, the software architecture embodies the integration of "monitoring-decision-execution" into the software stack of a distributed inference system, achieving closed-loop optimized control.

[0086] In this embodiment, an exemplary large language model inference service for online customer service Q&A is deployed. The model size can be 100 billion parameters, using FP16 weights and occupying approximately 200GB of video memory. The cluster hardware consists of 4 servers: two equipped with the latest NVIDIA H100 GPUs (8 GPUs each, 80GB of video memory per GPU, interconnected by NVSwitch high speed), and two equipped with slightly older Tesla A100 GPUs (8 GPUs each, 40GB of video memory per GPU) connected via 100Gb Ethernet, which does not support RDMA. The entire cluster has a total of 32 GPUs, with a total video memory resource of approximately 2×8×80 + 2×8×40) = 1792GB. However, the performance and memory of GPUs vary significantly between different nodes, and there is a bottleneck in network communication speed between old and new nodes. Communication within the H100 node is extremely fast, while communication between the H100 and A100 nodes is limited by Ethernet latency.

[0087] First, the agent's initial configuration. When the large language model inference system starts, the agent can scan the aforementioned heterogeneous environment: the two H100 nodes have high-bandwidth interconnection via NVLink / NVSwitch, suitable for cross-GPU parallelism; the two A100 nodes have limited GPU memory and slow cross-node communication. Based on this, the decision engine formulates an initial model partitioning scheme: a hybrid parallel strategy, deploying 70% of the model's parameters on the H100 nodes and 30% on the A100 nodes. Within each H100 node, 4-way tensor parallelism is employed (8 H100 GPUs per node are divided into two groups for parallel computation). Tensor parallelism is also performed between two H100 nodes (utilizing the high bandwidth of an RDMA-cascaded NVSwitch network), enabling the H100 cluster to collaboratively handle most of the model's computation. The A100 nodes, due to their smaller GPU memory, utilize a combination of pipelined parallelism and tensor parallelism. Each A100 node has four GPUs handling several layers of the later stages of the model, with each of the two A100 nodes completing a portion of the model pipeline. Finally, the results are aggregated on the H100 side. The first 70% of the model's layers primarily run on the H100 cluster, utilizing high computing power and bandwidth, while the last 30% runs on the A100 cluster, with lower computing power and bandwidth but smaller model fragments. Simultaneously, the agent identifies the network link without RDMA support between the H100 and A100 nodes, reducing communication in this segment. The decision engine schedules a small number of activation transfers, compressing or reducing the frequency of data crossing this network segment. Instead of synchronizing across nodes at every layer, intermediate results are only exchanged once during the pipeline stage. For the initial scheduling strategy of the initial orchestration scheme, the agent can place the main data-parallel shard on the H100 side, meaning most concurrent requests are processed first on the H100 cluster. If the request text is short and the later-stage computations of the model are not heavy, pipeline instances on the A100 cluster can maintain low utilization or remain dormant to avoid slowing down the overall process. In this way, through hardware awareness and heuristic strategies, the agent generates an initial orchestration scheme that matches the advantages of heterogeneous resources.

[0088] Secondly, consider the scenario of the large language model inference execution phase. After the system goes live, user questions are typically of moderate length (dozens of characters), and the model's responses are also moderate (around a hundred characters). In this typical scenario, the load is relatively balanced between the pre-filling and decoding phases. The H100 cluster handles most of the pre-filling layers and a portion of the decoding, while the A100 cluster handles the decoding computation for the last few layers. The two are connected via a pipeline; each pre-filling output is sent to the A100 to complete the final few layers of decoding before returning the result. Since most of the computation is done on the H100 and the H100 has strong performance, even though the A100 is relatively slower, it will not become a bottleneck. At the same time, the H100 cluster has high utilization, while the A100 cluster has lower utilization, but because the default scheduling prioritizes most requests to be executed on the H100, it can easily handle the situation. Therefore, the overall system latency and throughput are within acceptable limits.

[0089] Secondly, if the system encounters a special scenario and detects information about events requiring optimization, such as a trending topic or issue causing user questions to become lengthy (e.g., requiring AI to summarize a long article while simultaneously providing a detailed, lengthy answer), a surge in long input requests causes a sharp increase in computation during the pre-filling phase, and the decoding phase also needs to generate even longer text. At this point, the agent's perception layer detects that the utilization rate of the H100 cluster (the GPU responsible for pre-filling) is continuously approaching 100%, with requests waiting for pre-filling processing accumulating and the average response time significantly increasing. Conversely, while the A100 cluster is slower, it originally had fewer tasks. Now, the decoding phase's utilization rate has increased somewhat due to the longer answers, but it is still not at full capacity. Based on this, the agent triggers the event "pre-filling phase overload" and identifies that "the decoding GPU still has spare capacity." The decision engine determines to temporarily adjust resource allocation and increase pre-filling computing power. Since the H100 is already at full capacity, the most direct measure is to allocate some A100 resources to the pre-filling phase. The decision engine in the decision layer analyzes the rules base and uses optimization algorithms to determine the optimal orchestration scheme: "When pre-padding is overloaded and decoding is idle, a portion of the decoding pipeline can be converted to pre-padding." Based on this optimized orchestration scheme, the agent can redirect half of the GPUs on each A100 node that previously executed the last few layers of decoding to perform pre-padding computations on some of the earlier layers. This is equivalent to adjusting the pipeline splitting point: previously, the first 70% of the model's layers were executed on H100; now, the first 60% are executed on H100, the next 10% are executed on the freed-up A100 GPUs, and then the remaining 30% are returned to the other half of the A100 GPUs. This resharding and scheduling adjustment is dynamically completed through the framework's distributed loading interface—H100 stops executing the last few layers in pre-padding, sends the parameters of these layers to the designated A100 GPUs, and the task scheduler updates, forwarding the computation of the excessively long input to A100, then connecting back to H100 or continuing directly to the end of the decoding process. After this adjustment, the pre-filling load on the H100 cluster decreased by approximately 15%, and the pre-filling queue latency decreased. The A100 handled some pre-filling computations; since these GPUs were already waiting for longer outputs to be generated, the increased utilization did not lead to more queuing. Ultimately, the agent observed an overall system throughput increase of approximately 20% and a 10%–15% reduction in the average response latency for long requests through the monitoring layer, achieving a better service level. After the peak period, when the input length returned to normal, the agent, through its acquired hardware awareness information, detected idle pre-filling threads on the H100 and some idle pre-filling threads on the A100. This triggered a reverse event, migrating the model layer back to the H100 for execution, restoring the original division of labor, and allowing the A100 resources to return to a decoding-centric state. This process was completely automated; operations personnel only needed to view the agent's decisions and effects through monitoring logs, without manual intervention.

[0090] Here, in the aforementioned application scenarios, the adaptive agent orchestration scheme of this embodiment is compared with the traditional static orchestration scheme: In normal scenarios, the performance of both is similar, with the agent orchestration slightly better because its initial configuration has been optimized. The average utilization rate of the H100 cluster is 80%, and that of A100 is about 30%, with an average cluster throughput of, for example, reaching 50 requests / second and a P95 latency of about 200 milliseconds. In special long text scenarios, the traditional static orchestration scheme, due to lack of adjustment, causes continuous overload of H100, resulting in a throughput drop to 30 req / s and a P95 latency soaring to 500ms. In contrast, the scheme of this invention quickly adjusts resources after detecting load changes, causing the throughput to drop only slightly to 40 req / s and the P95 latency to remain around 300ms, significantly better than the unadjusted situation. Dynamic agent tuning can effectively maintain system performance stability and improve resource utilization.

[0091] Understandably, during the execution of a large model inference service, from input inference request to model output, data primarily flows as follows: User requests first reach the scheduling module, are encapsulated into inference tasks and placed in a queue. The Scheduler selects a group of GPUs to execute the task according to the current strategy. The task is broken down into sub-computations and sent to the execution engines on the corresponding GPUs. GPUs exchange intermediate results or synchronization states through communication modules, ultimately obtaining the model output, which is returned to the user. Simultaneously, a control data flow occurs. The agent collects hardware-aware information from various stages and makes decisions on orchestration schemes. These decisions generate a control command flow, which, through modules such as model management / scheduling, acts on the aforementioned main data flow, changing its direction or processing method. These two flows complement each other: the former represents actual business data, while the latter is control data that guides and optimizes the former.

[0092] It is understood that the solution in this embodiment can be an intelligent agent architecture based on a dynamic event perception and triggering mechanism. Its basic principle is to introduce an intelligent orchestration agent into the distributed inference system, continuously monitoring the system environment and runtime state, treating them as perceptible event sources, and responding to and triggering key events according to pre-set strategies or rules, thereby dynamically adjusting the configuration of the inference workflow. This intelligent agent integrates a closed-loop control mechanism of perception-decision-execution. The intelligent agent collects multi-level information in real time, including static environmental information and dynamic operational information. This collected information is filtered and analyzed, and abstracted into a series of events, such as "GPU memory usage exceeds a threshold," "Pre-filling stage processing latency is too high," "RDMA availability is detected," and "New request input length is abnormally long." For the perceived events, the intelligent agent's built-in decision engine formulates corresponding adjustment plans, i.e., optimizes the orchestration scheme, based on predetermined optimization algorithm logic. The core of this process is mapping environmental states to orchestrated actions. For example, if GPU memory is detected to be running low, the decision engine might trigger actions such as "model slicing repartitioning" or "enabling CPU memory caching." If the network lacks RDMA and cross-node communication is high, it might reduce model parallelism or adjust slicing to concentrate more computation on a single node. Furthermore, if the pre-filling GPU load is perceived to be significantly higher than the decoding GPU load, the decision engine might consider redirecting some decoding GPUs to pre-filling computations or temporarily switching to traditional single-node execution to balance the load. The decision-making process also comprehensively considers system-defined performance goals, such as maximizing throughput and latency SLO, as well as constraints, such as hardware resource limits and maintaining accuracy, ensuring that adjustments move towards the optimization goal while meeting constraints. Once the decision engine selects an optimization scheme, the agent translates the decision into control commands for the underlying inference workflow through an internal triggering mechanism, executing the adjustments immediately or gradually. This triggering mechanism is similar to a publish-subscribe control bus: the agent sends control signals to relevant modules, such as the model slicing management module, scheduler, and communication management module, instructing them to perform related operations. Performing related operations may include: reshaping the model's distribution across GPUs, such as migrating certain layers to idle GPUs, adjusting the tensor parallel partitioning ratio, modifying task scheduling strategies, such as changing the GPU bindings of pre-filling and decoding threads, or increasing the number of parallel instances, loading specific acceleration library paths, such as switching the TensorRT optimization kernel on key operators, etc. During the execution process, the agent can apply changes in a non-disruptive or minimally disruptive manner to reduce the impact on currently served requests. For example, it can wait for the current batch to complete before reallocating the model, or apply a new strategy to new requests while continuing to complete old requests according to the old strategy, ensuring a smooth system transition.

[0093] Through the aforementioned perception-decision-execution closed loop, the agent in this embodiment can act as an autonomous proxy, automatically reading hardware and software environment characteristics during deployment and selecting the initial optimal workflow orchestration scheme. During operation, it can continuously monitor performance indicators and adjust based on event triggers, ensuring the system always adapts to the current load and resource status. This method achieves adaptive optimization of the inference process: when the environment or load changes, the agent can optimize resource allocation and model partitioning in real time without manual intervention, avoiding performance degradation due to fixed configurations not matching new situations. In principle, this invention draws on the feedback regulation concept in cybernetics, abstracting the distributed inference system into a monitorable and controllable object, and dynamically optimizing and controlling it through the agent as a control unit, thereby solving the problem that traditional static schemes cannot accommodate changing conditions.

[0094] Thus, by adopting the solution in this embodiment, the intelligent agent can dynamically optimize resource allocation based on operational status, preventing unit bottlenecks from dragging down the overall system. When overloaded at a certain stage, resources are promptly expanded or parallelism is adjusted to prevent queue backlog; when resources are idle, tasks are reallocated to avoid waste. This adaptability ensures that all GPUs work together as efficiently as possible, significantly improving average throughput. Dynamically allocating GPU resources can increase the number of large model inference requests processed several times, and by balancing pre-filling / decoding loads, it also achieves a throughput improvement of over 20% and more stable low-latency performance. Compared to static configuration, which requires margin to prevent extreme situations, the intelligent agent's scheduling is more flexible, allowing the system to operate in a high utilization range under most normal conditions, providing a continuous performance advantage.

[0095] Furthermore, the solution described in this embodiment can automatically adapt to different hardware conditions through hardware capability awareness. For example, when RDMA is detected as available, a high-frequency, small-batch communication mode is used to reduce latency; conversely, communication is merged as much as possible to reduce overhead. Another example is selecting an appropriate degree of parallelism and whether to enable CPU memory participation based on GPU memory size. The agent can effectively utilize the respective advantages of the A100 and V100 hybrid cluster, making model deployment more efficient. Optimization can be performed through hardware awareness; regardless of environmental differences, the agent can adjust its strategy in real time to maximize the potential of existing hardware and reduce performance degradation caused by hardware bottlenecks.

[0096] Furthermore, in practical applications, inference requests can exhibit diverse characteristics, ranging from short and long texts to simple question-and-answer sessions or complex dialogues, and multi-user concurrency. A fixed inference pipeline struggles to handle all situations simultaneously. For example, simple question-and-answer sessions might only require smaller model shards for parallel processing to meet latency requirements, while long text summarization necessitates full model parallelism and more memory. The solution presented in this embodiment allows for dynamic switching of optimization strategies based on different load patterns, triggered by events. When an increase in the proportion of long-sequence requests is detected, the system automatically allocates more resources to pre-filling and adjusts the caching strategy; when requests become shorter and more numerous, it automatically favors small-batch, fast parallel processing. This ensures that regardless of workload changes, the service operates in a manner best suited to current needs, avoiding the dilemma of traditional solutions where peak-hour configurations lead to idle resources during normal times, or where normal-time optimizations become overwhelmed during peak periods, thus improving adaptability to complex workloads.

[0097] Furthermore, by employing the scheme described in this embodiment, the agent can continuously monitor resource usage, enabling the system to complete tasks with fewer resources while still meeting performance requirements. For example, when certain GPUs are under low load for extended periods and are not on critical paths, the agent can consider reducing parallelism or temporarily releasing these GPUs into standby mode, concentrating the load on other GPUs to improve single-card utilization and save energy. Conversely, when the load increases, more GPUs can be quickly activated for computation. This on-demand scaling strategy reduces unnecessary power consumption and rental costs, improving economic efficiency. Moreover, optimizing communication to reduce unnecessary data transmission also lowers network device load and energy consumption. In resource-constrained edge computing scenarios, the agent's intelligent scheduling allows critical computing power to be used for important parts, effectively expanding available resources.

[0098] Furthermore, using the scheme in this embodiment, the agent can acquire the real-time operating status of various parts of the system. In the event of anomalies such as GPU crashes or communication link interruptions, it can detect the event and take appropriate measures. For example, when a GPU is detected to be unresponsive, the agent can immediately remove it from the resource pool, the scheduling module can skip that GPU and assign new tasks, and resend unfinished tasks to other GPUs to ensure uninterrupted service. Another example is reducing cross-node dependencies during network anomalies and performing inference locally as much as possible to lower the probability of failure due to network problems. Compared to the rigid configuration of traditional solutions, the agent's dynamic adjustment capability makes the system more resilient. When encountering faults, it can self-degrade rather than completely crash, thereby improving service availability and enhancing system reliability while optimizing performance.

[0099] Furthermore, by employing the solution described in this embodiment, the intelligent agent significantly reduces the need for manual intervention through automatic perception and algorithmic decision-making. Deployers only need to provide basic optimization objectives, such as prioritizing latency or throughput and necessary constraints, and the intelligent agent can autonomously find the optimal configuration during operation. Especially when applications require frequent model updates or cluster expansion, it eliminates the workload of repetitive tuning each time. It reduces the need for engineers to manually adjust parameters such as model sharding schemes, parallelism, batch size, and caching strategies for different deployment environments and application scenarios, improving development and operational efficiency and making the management of complex large-model inference systems simpler and more intelligent.

[0100] Furthermore, by adopting the scheme in this embodiment, the autonomous agent control mechanism can be introduced into the distributed reasoning domain, overcoming the limitations of previous optimization techniques that operated independently and were statically configured. Through global observation and real-time response, the system can act like an experienced driver, adjusting the accelerator and brake according to road conditions, rather than mechanically maintaining a fixed speed like cruise control. This dynamic closed-loop optimization approach enables the system to continuously correct its behavior based on feedback, consistently approaching its optimal performance.

[0101] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0102] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0103] Figure 4 This invention provides a structural block diagram of a large model inference optimization apparatus according to an embodiment of the present application, as shown below. Figure 4As shown. The large model inference optimization device 400 of this embodiment is applied to an intelligent agent and may include an acquisition unit 401, a determination unit 402, an acquisition unit 403, and an optimization unit 404. The acquisition unit 401 is used to acquire hardware awareness information and inference task information of the large model inference system, the hardware awareness information including static environment information and dynamic operation information; the determination unit 402 is used to determine the event information to be optimized based on preset event triggering conditions, the inference task information, and the hardware awareness information, using a hardware evaluation strategy; the acquisition unit 403 is used to obtain an optimized orchestration scheme based on the preset knowledge base and the event information to be optimized, using an optimization algorithm; the optimization unit 404 is used to optimize the large model inference system based on the optimized orchestration scheme.

[0104] Optionally, in one possible implementation of this embodiment, the acquisition unit 401 is used to acquire the static environment information in response to the inference service startup of the large model inference system; acquire the dynamic operation information in response to the large model inference system performing inference service based on the received inference request and initial orchestration scheme; and acquire inference task information based on the inference request received by the large model inference system.

[0105] Optionally, in one possible implementation of this embodiment, the determining unit 402 is configured to determine event information that satisfies the preset event triggering conditions based on the preset event triggering conditions and the hardware perception information; evaluate the event information that satisfies the preset event triggering conditions using a hardware evaluation strategy based on the inference task information; and determine the event information to be optimized based on the result of the evaluation process.

[0106] Optionally, in one possible implementation of this embodiment, the determining unit 402 is used to perform noise reduction processing on the result of the evaluation process; and determine the event information to be optimized based on the result of the noise reduction processing.

[0107] Optionally, in one possible implementation of this embodiment, the obtaining unit 403 is used to determine the constraints and heuristic information of the optimization algorithm based on the preset knowledge base, so as to construct the optimization algorithm; based on the event information to be optimized, use the optimization algorithm to obtain the expected benefits of the candidate orchestration schemes corresponding to the event information to be optimized; and based on the expected benefits of the candidate orchestration schemes, evaluate the candidate orchestration schemes to obtain the optimized orchestration scheme.

[0108] Optionally, in one possible implementation of this embodiment, the optimization unit 404 is used to adjust the current orchestration scheme of the large model inference system based on the optimized orchestration scheme; obtain the hardware awareness information of the large model inference system after adjustment; evaluate the hardware awareness information after adjustment; and, in response to the result of the evaluation not meeting the preset evaluation conditions, revert the orchestration scheme of the large model inference system to the current orchestration scheme before adjustment.

[0109] In this embodiment, the hardware awareness information and inference task information of the large model inference system can be acquired by the acquisition unit. The hardware awareness information includes static environment information and dynamic operation information. The determination unit, based on preset event triggering conditions, the inference task information, and the hardware awareness information, uses a hardware evaluation strategy to determine the event information to be optimized. The acquisition unit, based on the preset knowledge base and the event information to be optimized, uses an optimization algorithm to obtain an optimized orchestration scheme. The optimization unit optimizes the large model inference system based on the optimized orchestration scheme. Since the acquired hardware awareness information and inference task information can be processed... Hardware usage assessment dynamically identifies events requiring optimization. Optimization algorithms then generate corresponding orchestration schemes for these events. These schemes effectively optimize the current orchestration of the large-scale model inference system, enabling adaptive orchestration optimization of the workflow in distributed large-scale model inference. This dynamic adjustment of the inference workflow orchestration scheme overcomes the limitations of traditional static configuration, dynamically optimizing resource allocation based on operational status. This avoids unit bottlenecks hindering the overall system, improves average throughput, optimizes latency, and enhances resource utilization during system operation, thereby ensuring the performance and stability of the distributed large-scale model inference system.

[0110] The technical solution of this application involves the collection, storage, use, processing, transmission, provision, and disclosure of user personal information, such as user image and attribute data, which comply with relevant laws and regulations and do not violate public order and good morals.

[0111] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0112] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0113] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0114] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the method of large model inference optimization. For example, in some embodiments, the method of large model inference optimization may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the method of large model inference optimization described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method of large model inference optimization by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0121] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0122] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for optimizing large model inference, characterized in that, The method includes: Acquire hardware awareness information and inference task information of a large model inference system, wherein the hardware awareness information includes static environment information and dynamic operation information; Based on preset event triggering conditions, the inference task information, and the hardware perception information, the event information to be optimized is determined using a hardware evaluation strategy. Based on the preset knowledge base and the event information to be optimized, an optimized orchestration scheme is obtained using an optimization algorithm; Based on the optimized orchestration scheme, the large model inference system is optimized.

2. The method according to claim 1, characterized in that, The acquisition of hardware awareness information and inference task information of the large model inference system includes: In response to the startup of the inference service of the large model inference system, the static environment information is obtained; In response to the large model inference system performing inference services based on the received inference request and initial orchestration scheme, the dynamic operation information is obtained; Based on the inference request received by the large model inference system, inference task information is obtained.

3. The method according to claim 1, characterized in that, Based on preset event triggering conditions, inference task information, and hardware awareness information, a hardware evaluation strategy is used to determine the event information to be optimized, including: Based on the preset event triggering conditions and the hardware sensing information, determine the event information that satisfies the preset event triggering conditions; Based on the inference task information, a hardware evaluation strategy is used to evaluate and process event information that meets the preset event triggering conditions. Based on the results of the evaluation process, the event information to be optimized is determined.

4. The method according to claim 3, characterized in that, Based on the results of the evaluation process, the information on events to be optimized is determined, including: The results of the evaluation process are then denoised. Based on the results of the denoising process, the event information to be optimized is determined.

5. The method according to claim 1, characterized in that, The step of obtaining an optimized orchestration scheme based on the preset knowledge base and the event information to be optimized using an optimization algorithm includes: Based on the preset knowledge base, the constraints and heuristic information of the optimization algorithm are determined to construct the optimization algorithm; Based on the event information to be optimized, the expected benefits of the candidate orchestration schemes corresponding to the event information to be optimized are obtained using the optimization algorithm. Based on the expected benefits of the candidate orchestration schemes, the candidate orchestration schemes are evaluated to obtain the optimized orchestration scheme.

6. The method according to claim 1, characterized in that, The optimization process for the large-scale model inference system based on the optimized orchestration scheme includes: Based on the optimized orchestration scheme, the current orchestration scheme of the large model inference system is adjusted. Obtain the adjusted hardware perception information of the large model inference system; The adjusted hardware perception information is evaluated and processed. If the result of the evaluation process does not meet the preset evaluation conditions, the orchestration scheme of the large model inference system is reverted to the current orchestration scheme before the adjustment process.

7. An apparatus for optimizing large-scale model inference, characterized in that, The device includes: The acquisition unit is used to acquire hardware perception information and inference task information of the large model inference system. The hardware perception information includes static environment information and dynamic operation information. The determining unit is used to determine the event information to be optimized based on preset event triggering conditions, the inference task information, and the hardware perception information, using a hardware evaluation strategy. The obtaining unit is used to obtain an optimized orchestration scheme based on the preset knowledge base and the event information to be optimized, using an optimization algorithm. The optimization unit is used to optimize the large model inference system based on the optimized orchestration scheme.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images