Dynamic reasoning load balancing service method for doctor-patient dialogue model

Abstract

The invention belongs to the technical field of large language model services, and particularly relates to a dynamic reasoning load balancing service method for a doctor-patient dialogue model. According to the method, a collaborative optimization end-to-end service framework is constructed, and the core of the collaborative optimization end-to-end service framework is a concurrent sensing dynamic routing mechanism; the mechanism can monitor the amount of concurrent requests of doctor-patient dialogues in real time, and intelligently route the requests to heterogeneous reasoning replica clusters with different parallel (such as tensor parallel and replica parallel) strategies, so that the problem that when an existing service architecture faces a low-concurrency and high-concurrency mixed load, the concurrency of the doctor-patient dialogues is influenced is effectively solved. The problem of resource waste or request blocking caused by adopting static single configuration is solved, and elastic and efficient utilization of computing resources is achieved. Experimental results prove that the effect of the method is remarkably better than that of the existing technical method in the aspects of improving the overall throughput of a doctor-patient dialogue system, reducing the average waiting time of the users and supporting simultaneous online consultation of larger-scale users.

Description

Technical Field

[0001] This invention relates to the field of large-scale language model service technology, and in particular to a dynamic inference load balancing service method for doctor-patient dialogue models. Background Technology

[0002] In recent years, large language models (LMs) have demonstrated enormous application potential in the healthcare field, particularly in simulating doctor-patient dialogue scenarios such as initial consultations, health advice, medication reminders, and post-treatment follow-ups. These doctor-patient dialogue models can provide services 24 / 7, significantly alleviating the strain on healthcare resources. To ensure a positive user (i.e., patient) experience, the service system must be able to respond to user inquiries with near real-time, low-latency responses. Any noticeable delay can lead to user anxiety and distrust of the service. Therefore, how to efficiently and stably provide low-latency dialogue model inference services to a large number of concurrent users is a key technical challenge for the large-scale commercial application of this technology.

[0003] The core of this task is to build a large model service system capable of handling high-concurrency requests while maintaining low end-to-end latency. End-to-end latency includes not only the time required for the model itself to perform computation and inference (i.e., inference engine latency), but also the total time spent on the entire chain, from the user's request being sent, through network transmission, reception by the service gateway, queuing, routing, to the return of the inference result to the user.

[0004] Currently, the technical solutions in this field can be mainly divided into two categories: optimization techniques focusing on the performance of the inference engine itself; and building service gateways using general-purpose web frameworks. Optimization techniques focusing on the performance of the inference engine itself are the mainstream research direction. These techniques aim to accelerate the speed of single or batch inference of models on computing devices (such as GPUs). Specific methods include: Model parallelization: such as tensor parallelism, which splits the model's computational tasks across multiple GPUs to complete collaboratively, accommodating models with a large number of parameters; Memory management optimization: such as the PagedAttention mechanism, which improves GPU utilization and system throughput by efficiently managing the key-value cache in attention computation, reducing memory fragmentation and unnecessary copying; Computation process optimization: such as FlashAttention, which reduces the number of GPU memory reads and writes through operator fusion; Model quantization, which reduces memory usage and accelerates computation by reducing the precision of model parameters (e.g., from FP16 to INT8); Request scheduling optimization: such as continuous batch processing, which allows new requests to be dynamically added to the batch while the GPU is processing the current batch of tasks, avoiding GPU idleness and improving processing efficiency. In the construction of service gateways using general web frameworks, the service gateway serves as the entry point for the entire service system, responsible for receiving all user requests and distributing them to the backend inference engine. Developers typically use mature web frameworks, such as Python-based FastAPI, to quickly build this module due to its ease of development and mature ecosystem. However, the aforementioned existing technical solutions exhibit significant bottlenecks and limitations when dealing with large-scale, high-concurrency doctor-patient dialogue scenarios. The core issue lies in the lack of end-to-end, systematic optimization of the service chain. Specifically, optimization techniques focused on the performance of the inference engine itself limit the optimization target to the inference engine itself, representing a form of "local optimization." While these techniques significantly improve the computation speed of the model, they ignore the fact that when the number of concurrent requests increases dramatically, the service gateway becomes a new and more significant performance bottleneck. Under high concurrency (for example, a city-level online consultation platform may face thousands or even tens of thousands of QPS during peak periods), the service gateway needs to handle intensive network I / O, request parsing, security verification, load balancing decisions, and other tasks. FastAPI, a service gateway based on a general Python framework, suffers from limitations in its underlying mechanisms (such as the Global Interpreter Lock, GIL) that make it difficult to fully utilize the server's multi-core CPU resources for efficient processing of these tasks. This leads to a large backlog of requests at the gateway, resulting in severe latency. At this point, even if the backend inference engine has a high computation speed, it cannot receive the necessary requests in time, and the overall system response speed remains slow. Experimental data shows that in high-concurrency scenarios, gateway latency can even exceed inference engine latency, becoming the main component of end-to-end latency.

[0005] In summary, existing technologies only optimize the inference engine at a "point" level, while ignoring the bottleneck of the service gateway under high concurrency. There is a lack of an end-to-end comprehensive solution that can collaboratively optimize the gateway and engine and dynamically adjust the system architecture according to real-time load. As a result, in actual deployment of doctor-patient dialogue services, the system is unable to effectively scale to serve a large number of concurrent users while meeting low latency requirements, which greatly limits the application scope and user experience of this technology. Summary of the Invention

[0006] To address the technical problems existing in the current doctor-patient dialogue model service architecture, namely that existing solutions only optimize the computation speed of the inference engine while ignoring the fact that the service gateway becomes the main bottleneck of system performance under high concurrency requests, resulting in high response latency and low throughput of the entire service chain under high load, the present invention aims to propose an end-to-end service method for doctor-patient dialogue models that can collaboratively optimize the service gateway and inference engine and dynamically route according to real-time load.

[0007] This invention addresses the fundamental issue of insufficient processing capacity of traditional Python gateways under high concurrency by constructing a high-performance, asynchronous service gateway. Furthermore, it introduces a concurrency-aware dynamic routing mechanism that intelligently distributes requests to heterogeneous inference replica clusters employing different parallel strategies based on real-time changes in concurrent request volume. This dynamically adaptive architecture overcomes the limitations of traditional service systems that use static, single configurations, leading to either wasted resources under low load or drastically reduced performance under high load when facing complex and variable doctor-patient consultation loads. Extensive experiments have demonstrated that this invention significantly outperforms existing technologies in improving the overall throughput of doctor-patient dialogue systems and significantly reducing the average user waiting time.

[0008] The technical solution of the present invention is described in detail below.

[0009] A dynamic inference load balancing service method for doctor-patient dialogue models includes the following sequentially executed steps: Step 1, Concurrency Load Monitoring Phase: Monitor and count the number of concurrent requests for the current doctor-patient dialogue service in real time through the service gateway to quantify the current real-time load level of the system; Step 2, Optimal Inference Replica Selection Phase: The number of concurrent requests is used as input and sent to a dynamic routing strategy module. This module selects the inference replica or replica group that best matches the current load level from a cluster composed of heterogeneous inference replicas based on one or more preset load thresholds. Step 3: Request Routing and Execution Phase: The service gateway forwards the doctor-patient dialogue request of the new user to the inference replica or replica group selected in Step 2 for processing, and returns the generated dialogue result to the user, thus completing a complete end-to-end service process.

[0010] In the present invention, in Step 2, the cluster composed of heterogeneous inference replicas is configured as follows: it includes at least one type of inference replica configured with a low replica number and a high tensor parallelism, aiming to minimize the single response latency when processing small-batch requests by concentrating computing resources. This is the first type of inference replica; and it includes at least another type of inference replica configured with a high replica number and a low tensor parallelism, aiming to disperse and process large-scale concurrent requests through replica parallelism to maximize the total throughput of the system. This is the second type of inference replica.

[0011] In the present invention, in Step 2, the selection method of the dynamic routing policy module is based on a preset concurrency threshold C for decision-making: When the monitored number of concurrent requests N is lower than the threshold C, it is determined that the system is in a low concurrency state, and the first type of inference replica is selected; When the monitored number of concurrent requests N is equal to or higher than the threshold C, it is determined that the system is in a high concurrency state, and the second type of inference replica is selected.

[0012] In the present invention, the dynamic routing policy module defines at least two different load thresholds C₁ and C₂, where C₁ < C₂, thus dividing the system load into at least three intervals: low, medium, and high, and configuring a specific inference replica group with different combinations of replica numbers and tensor parallelisms for each interval. Specifically, on the premise of a constant total computing resource (such as the total number of GPUs), the configuration combination is achieved by changing the product combination of the replica number and the tensor parallelism of each replica; low load interval: aiming to minimize latency, a combination of high tensor parallelism TP and low replicas should be selected; high load interval: aiming to maximize throughput, a combination of low tensor parallelism TP and high replicas should be selected; medium load interval: as a transition, a combination between the two is selected.

[0013] In the present invention, in Steps 1 and 3, the service gateway for receiving and forwarding requests is built using an asynchronous runtime framework based on the Rust language to utilize its fearless concurrency characteristics and efficient I / O processing capabilities to support大规模并发的网络连接。

[0014] In the present invention, the asynchronous runtime framework is Tokio, and the service gateway is implemented based on the Axum Web framework.

[0015] In this invention, the service gateway and the heterogeneous inference replica communicate using the gRPC protocol, and a reusable gRPC connection pool is used to manage and schedule connections, thereby reducing the system overhead and latency caused by frequent connection establishment and destruction.

[0016] In this invention, the gRPC protocol uses Protocol Buffers as its interface definition language and underlying data serialization format to achieve efficient, language-independent data exchange.

[0017] In this invention, in step three, the optimization techniques used within the inference copy are selected from a set of the following techniques: Model quantization: Converting the floating-point parameters of the model into low-bit integers or half-precision floating-point numbers to accelerate computation; PagedAttention mechanism: Efficiently manages the key-value cache in attention computation through virtualized memory management; FlashAttention: Reduces GPU memory read / write operations through operator fusion technology; Continuous batch processing: Allows new requests to be added dynamically to the processing batch during inference.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Solves gateway performance bottleneck and achieves true end-to-end optimization: This invention not only optimizes the inference engine, but also fundamentally solves the problem of gateways becoming performance bottlenecks under high concurrency in traditional solutions by building a service gateway using the high-performance Rust asynchronous framework and gRPC communication protocol. This global, end-to-end optimization of the service chain ensures that the system can achieve maximum performance under any load.

[0019] 2. Achieves elastic and adaptive resource utilization: This invention, through an innovative dynamic routing mechanism, enables the service architecture to adapt its service form like a "Transformer" based on real-time load. During off-peak periods for patient consultations, the system automatically switches to a high-tensile parallel mode to conserve and efficiently utilize resources; during peak periods, it automatically switches to a high-replica parallel mode to handle massive concurrent requests. This elasticity overcomes the limitations of static configuration, achieving a dynamic balance between resource utilization and system performance.

[0020] Based on the above two points, this invention demonstrates extremely high throughput and extremely low average latency when facing large-scale concurrent doctor-patient dialogue requests. Experimental results show that compared with existing advanced technologies that only optimize the inference engine, this invention can achieve several times the performance improvement in high-concurrency scenarios. This means that the system can serve more users online simultaneously, and users' waiting time is shorter, greatly improving the experience of online doctor-patient communication. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall architecture of the dynamic inference load balancing service system for the doctor-patient dialogue model proposed in this invention.

[0022] Figure 2 This is a logical flowchart of the core mechanism of this invention—the concurrency-aware dynamic routing strategy. Detailed Implementation

[0023] The technical solution proposed in this invention will now be described in a comprehensive and detailed manner with reference to the accompanying drawings and embodiments.

[0024] While existing solutions for doctor-patient dialogue model services have made significant progress at the inference engine level by introducing advanced frameworks such as vLLM, they generally suffer from a "local optimum" design flaw. At the system architecture level, these solutions still follow the traditional model of using a general-purpose web framework (such as Python FastAPI) as the service gateway. The inherent problem with this architecture is its failure to consider the service chain from end to end systematically. Its core shortcomings are twofold: First, the processing capacity of the service gateway itself has theoretical limits and practical bottlenecks under high concurrency, especially when constrained by the Python global interpreter lock, making it impossible to effectively utilize the multi-core computing power of modern servers; second, the configuration strategy for inference resources is static and fixed, unable to adaptively adjust to the load changes with distinct peak and trough characteristics common in doctor-patient consultation scenarios.

[0025] This invention aims to fundamentally solve the system performance problems caused by gateway bottlenecks and static resource configuration, and proposes an innovative, more precise, more efficient, and more flexible end-to-end service framework that is innovative at both the architecture and service strategy levels.

[0026] To achieve this goal, this invention introduces the core concepts of collaborative optimization and dynamic adaptation. Collaborative optimization is reflected in the simultaneous deep customization and performance matching of the front end (gateway) and back end (inference engine) of the service link; dynamic adaptation is reflected in the construction of a dynamic routing mechanism that can sense real-time load and intelligently switch back end resource configuration strategies.

[0027] like Figure 1 As shown, the overall framework of the present invention is that after receiving concurrent user requests 100, a high-performance service gateway 101 is used as the entry point, a dynamic routing module 102 is embedded as the decision brain, and it is connected to a heterogeneous inference replica cluster 103 composed of replicas with different parallel strategies, including first-type inference replicas and second-type inference replicas.

[0028] Figure 2This paper details the core mechanism of the invention: the internal working principle of the concurrency-aware dynamic routing strategy module. The process begins with real-time concurrency monitoring 200, with a key decision node 201 that compares the current number of concurrent requests N with a preset threshold C. Based on the comparison result, the process branches into two different paths: one to a low-concurrency routing path 202, and the other to a high-concurrency routing path 203, thus achieving intelligent request routing.

[0029] The specific implementation steps and details of the present invention are described in detail below.

[0030] Step 1: High-performance gateway request reception and concurrent monitoring phase This phase aims to build a robust front-end entry point that will not become a system bottleneck. The technical features of the high-performance service gateway 101 include: Underlying framework: It is developed using an asynchronous runtime framework based on the Rust language (such as Tokio and Axum) to fully utilize the server's multi-core CPU resources and efficiently handle massive network I / O requests.

[0031] Communication Protocol: The gateway communicates with the backend inference replicas using the gRPC protocol, and leverages reusable connection pools and efficient data serialization formats to minimize communication overhead.

[0032] The specific implementation details are as follows: Gateway Technology Selection and Construction: This invention preferably uses the Rust language and its ecosystem, including the Tokio asynchronous runtime and the Axum Web framework, to build a high-performance service gateway 101. Its technical advantages are: Ultimate concurrency handling capabilities: Rust's asynchronous model is based on non-blocking I / O, allowing a single thread to efficiently manage tens of thousands of concurrent connections, fundamentally solving the problem of thread resource exhaustion under the traditional synchronous blocking model.

[0033] Full utilization of hardware: Unlike Python, which is subject to the GIL, Rust applications can execute computationally intensive tasks (such as request parsing, authentication, logging, etc.) in parallel on all CPU cores without any burden, ensuring that the gateway remains responsive under high concurrency.

[0034] High-efficiency communication protocol: The communication between the high-performance service gateway 101 and the backend heterogeneous inference replica cluster 103 adopts the gRPC protocol.

[0035] Compared to traditional JSON-based RESTful APIs, gRPC utilizes Protocol Buffers for data serialization, resulting in a more compact binary format and faster parsing speed. Furthermore, its HTTP / 2-based multiplexing capabilities allow for the parallel sending and receiving of multiple requests and responses over a single TCP connection, significantly reducing connection establishment overhead and network latency.

[0036] Real-time concurrency monitoring: Inside this high-performance service gateway 101, the dynamic routing module 102 uses efficient algorithms such as atomic counters or sliding time windows to count the number of currently active concurrent requests N in real time with extremely low performance overhead, and uses it as the core input for the next decision.

[0037] Step 2: Concurrency Awareness and Dynamic Routing Decision Stage This stage is the "brain" of the invention, responsible for making optimal resource scheduling decisions based on real-time load.

[0038] 1. Core Decision Logic: The core of the dynamic routing module 102 is a threshold-based rule engine. The module compares the number of real-time concurrent requests N with one or more preset thresholds C. This threshold C is not arbitrarily set, but scientifically determined after offline stress testing and performance profiling of specific models and hardware, representing the "intersection point" of the performance curves of different parallel strategies.

[0039] 2. Strategy Selection Mechanism: An example using a single threshold C (e.g., C=64 as determined by testing) illustrates the decision-making process as follows: When N < 64, the system is determined to be in a low-concurrency state. At this time, decision node 201 will point to the low-concurrency routing path 202 optimized for this scenario.

[0040] When N ≥ 64, the system determines that it is in a high-concurrency state. At this time, decision node 201 will switch and direct subsequent requests to the high-concurrency routing path 203 optimized for this scenario.

[0041] Step 3: Heterogeneous Copy Inference Execution Phase This stage is the physical execution layer of the strategy of this invention, and its key lies in the "heterogeneous" copy configuration.

[0042] Trade-offs in parallel strategies: Tensor parallelism: This involves distributing the weight tensors of a large single model across multiple graphics processing units (GPUs). Its advantage is that it can accommodate massive models that a single GPU cannot handle. However, its disadvantage is the significant communication overhead caused by the need for high-frequency data synchronization (such as All-Reduce operations) between multiple GPUs during inference. Therefore, while tensor parallelism (TP) effectively reduces latency when processing single or small batches of requests, its overall throughput is limited by communication bottlenecks when a large number of independent requests flood in.

[0043] Replica Parallelism: Deploying multiple identical, independent replicas of the model on different GPUs (or groups of GPUs). Its advantage is that no communication is required between replicas, allowing for perfect parallel processing of multiple independent request streams. Its disadvantage is that it cannot handle excessively large single models.

[0044] The specific configuration of the heterogeneous replica group: Low-concurrency replica groups (optimized for minimum latency): These include Type 1 inference replicas, employing a strategy that maximizes tensor parallelism and minimizes replica parallelism. For example, on an 8-GPU server, this could be configured as {Replicas: 1, TP_degree:8}. This means all 8 GPUs work together to process one (or a small batch of) doctor-patient dialogue requests as quickly as possible, with the goal of minimizing perceived latency for individual users.

[0045] High-concurrency replica groups (optimized for maximum throughput): This includes a second type of inference replicas, employing a strategy of maximizing replica parallelism and minimizing tensor parallelism. For example, a single 8-GPU server might be configured as {Replicas: 4, TP_degree: 2}. This means the server is divided into four independent inference units, each with two GPUs. When a massive number of requests arrive, they can be evenly distributed across these four "service windows" for parallel processing, aiming to maximize the total number of users served per unit time (i.e., system throughput).

[0046] Example 1 To quantitatively verify the significant superiority of the method described in this invention compared to the prior art, we constructed a rigorous comparative experimental environment.

[0047] 1) Experimental Environment Setup Hardware platform: A server configured with 8 NVIDIA DGX H100 GPUs.

[0048] Software Model: The widely used open-source doctor-patient dialogue platform model, Mixtral 8×7B, was selected.

[0049] Load simulation: Using professional load generation tools, simulate 1 to 256 concurrent users continuously sending typical consultation requests.

[0050] (ii) Comparison Methods Baseline approach: Employing the industry-leading vLLM inference framework, paired with the commonly used Python FastAPI as the service gateway. This approach represents a high-performance "local optimization" solution in the current technology.

[0051] The method of this invention: It adopts the end-to-end optimization framework detailed in this paper, namely a high-performance gateway based on Rust / Axum / gRPC, and has a built-in concurrency-aware dynamic routing module. The backend is connected to a heterogeneous inference replica configured according to TP=8 and TP=2.

[0052] III) Evaluation Indicators and Experimental Results System throughput (tokens / s): A core metric for measuring system processing capacity.

[0053] Latency (s): A core metric for measuring user experience.

[0054] Table 1: Performance Comparison of the Invention with Existing Public Techniques

[0055] As can be seen from Table 1: In low-concurrency areas (such as 16 concurrent users), the performance difference between the two methods is not significant, and both can provide smooth service.

[0056] The performance watershed occurs around 64 concurrent connections. The baseline method's latency begins to climb sharply, and throughput growth slows significantly, clearly indicating that its FastAPI gateway is becoming saturated and a system bottleneck. However, the method of this invention triggers dynamic routing at this node, enabling high-concurrency replica groups, thus allowing its throughput to continue to surge while maintaining extremely low latency.

[0057] In high-concurrency regions (128 and 256 concurrent users), the performance difference between the two is significant. The throughput of the method in this invention is more than 2.3 times that of the baseline method, while the end-to-end latency is only about one-quarter of that of the baseline method.

[0058] To further illustrate the synergistic effects of this invention, Table 2 presents a set of ablation experiments. The data shows that when only the backend engine is optimized while the traditional gateway is retained (i.e., the "engine-only optimization solution"), under high concurrency, the performance bottleneck shifts to the frontend, resulting in an increase in end-to-end latency compared to the baseline, leading to performance degradation. This result conversely proves that this invention, through the collaborative work of a high-performance gateway and an optimized backend, solves the bottleneck shift problem caused by single optimization, thus producing a synergistic effect where 1+1 is far greater than 2.

[0059] Table 2: Comparison of ablation experiments showing the effects of synergistic optimization (high-concurrency zone)

[0060] Experimental results demonstrate that the implementation of the method of this invention significantly reduces the end-to-end average latency (by approximately 78%) of the entire doctor-patient dialogue service system under high-concurrency load conditions compared to systems using static, homogeneous replica configurations, while simultaneously significantly increasing the total system throughput (by more than 100%). This invention, through the collaborative innovation of a high-performance gateway and concurrency-aware dynamic routing, successfully transforms a static, bottlenecked service architecture into a resilient, end-to-end optimized, and advanced service system capable of handling large-scale concurrent requests, providing a solid technical guarantee for the implementation of serious AI applications such as doctor-patient dialogue.

Claims

1. A dynamic inference load balancing service method for doctor-patient dialogue models, characterized in that, The following steps are executed consecutively: Step 1, Concurrency Load Monitoring Phase: Monitor and count the number of concurrent requests for the current doctor-patient dialogue service in real time through the service gateway to quantify the current real-time load level of the system; Step 2, Optimal Inference Replica Selection Phase: The number of concurrent requests is used as input and sent to a dynamic routing strategy module. This module selects the inference replica or replica group that best matches the current load level from a cluster composed of heterogeneous inference replicas based on one or more preset load thresholds. Step 3, Request Routing and Execution Phase: The service gateway forwards the new user's doctor-patient dialogue request to the inference replica or replica group selected in Step 2 for processing, and returns the generated dialogue result to the user, thus completing a complete end-to-end service process.

2. The dynamic inference load balancing service method according to claim 1, characterized in that, In step two, the cluster composed of heterogeneous inference replicas is configured as follows: at least one type of inference replica is configured with a low number of replicas and high tensor parallelism, which aims to minimize the single response latency when processing small batch requests by concentrating computing resources. This is the first type of inference replica. And at least one other type of inference replica is included, configured with a high number of replicas and low tensor parallelism, designed to distribute and process large-scale concurrent requests through replica parallelism to maximize the total throughput of the system. This is the second type of inference replica.

3. The dynamic inference load balancing service method according to claim 2, characterized in that, In step two, the selection method of the dynamic routing strategy module is based on a preset concurrency threshold C: When the number of concurrent requests N detected is lower than the threshold C, the system is determined to be in a low-concurrency state, and the first type of inference replica is selected. When the number of concurrent requests N detected is equal to or higher than the threshold C, the system is determined to be in a high-concurrency state, and the second type of inference replica is selected.

4. The dynamic inference load balancing service method according to claim 3, characterized in that, The dynamic routing strategy module defines at least two different load thresholds C1 and C2, where C1 < C2, thereby dividing the system load into at least three intervals: low, medium, and high, and configuring inference replica groups with different combinations of replica count and tensor parallelism for each interval.

5. The dynamic inference load balancing service method according to claim 1, characterized in that, In steps one and three, the service gateway that receives and forwards requests is built using an asynchronous runtime framework based on the Rust language to leverage its fearless concurrency features and efficient I / O processing capabilities to support large-scale concurrent network connections.

6. The dynamic inference load balancing service method according to claim 5, characterized in that, The asynchronous runtime framework is Tokio, and the service gateway is implemented based on the Axum Web framework.

7. The dynamic inference load balancing service method according to claim 5, characterized in that, The service gateway communicates with the heterogeneous inference replica using the gRPC protocol, and manages and schedules connections through a reusable gRPC connection pool to reduce the system overhead and latency caused by frequent connection establishment and destruction.

8. The dynamic inference load balancing service method according to claim 7, characterized in that, The gRPC protocol utilizes Protocol Buffers as its interface definition language and underlying data serialization format to achieve efficient, language-independent data exchange.

9. The dynamic inference load balancing service method according to claim 1, characterized in that, In step three, the optimization techniques used within the inference copy are selected from a set of the following techniques: Model quantization: Converting the floating-point parameters of the model into low-bit integers or half-precision floating-point numbers to accelerate computation; PagedAttention mechanism: Efficiently manages the key-value cache in attention computation through virtualized memory management; FlashAttention: Reduces GPU memory read / write operations through operator fusion technology; Continuous batch processing: Allows new requests to be added dynamically to the processing batch during inference.

Images