Multi-model dynamic scheduling method for high-concurrency dialogue request

Abstract

The application belongs to the technical field of distributed large language model inference scheduling, and relates to a multi-model dynamic scheduling method for high-concurrency dialogue requests, comprising the following steps: collecting a network card hard interrupt timestamp sequence at a stream response bus to establish a stream output time interval vector; extracting a differential fluctuation envelope of the stream output time interval vector and comparing the differential fluctuation envelope with a benchmark transmission time delay line to separate a channel congestion index; calculating a real-time routing inhibition coefficient according to a queue backlog length and the channel congestion index, and reconstructing a cost-aware routing matrix; when concurrent data flow is overloaded, a sampling window is compressed, and when the real-time routing inhibition coefficient is out of limit, a redirection signal is sent to shift the concurrent data flow; the application does not need active detection, can decouple calculation congestion and network jitter in situ, effectively eliminates long tail response delay, and improves throughput.

Description

Technical Field

[0001] This invention belongs to the field of distributed large language model inference scheduling technology, and relates to a multi-model dynamic scheduling method for high-concurrency dialogue requests. It is used in the case of large-scale network burst traffic and cross-layer resource time-series coordination of large models, which can effectively eliminate long-tail response latency and improve system throughput. Background Technology

[0002] Currently, utilizing heterogeneous computing GPU clusters and high-performance networks for distributed large-scale language model inference has become a core approach in intelligent computing systems. However, existing technologies still have limitations in handling cross-layer resource timing coordination and physical boundary constraint awareness when dealing with large-scale network burst traffic.

[0003] First, there is the dynamic memory allocation technology based on task metadata awareness. For example, Chinese invention patent CN120295737A discloses a computing card group scheduling method and device based on task awareness and generation length prediction. It uses a lightweight BERT model to predict the generation length of user prompt words, and dynamically requests multi-granularity memory blocks from the shared memory pool based on the prediction results to generate memory block handles and write them into metadata. Based on this, a task scheduling sequence is constructed. Its shortcomings are: this scheme is essentially a computing card resource control within a single node. The decision logic lacks the linkage awareness of the transient congestion state of the entire network communication link. It does not map the temporal fluctuations of network-side data flow injection into the boundary constraints of the subsequent scheduling space. As a result, when facing large-scale concurrent network traffic impacts, it is easy to cause spatiotemporal mismatch of computing resources and occasional idle computing resources.

[0004] Second, there is the network card interrupt control technology based on host busyness. For example, Chinese invention patent CN115632948B discloses an interrupt control method, device, storage medium, and electronic device applied to network cards. By statistically analyzing the number of data packets received within a historical time window and determining the host busyness value, it uses an evaluation model to estimate the scale of data packets received within a future time window, and adjusts the adaptive interrupt strategy accordingly to control the timing of the network card initiating an interrupt request to the host. Its shortcomings are: this technology focuses on overhead control and open-loop prediction at the underlying hardware interface level, and fails to establish a closed-loop correlation between network card interrupt control and the real-time load status of upper-layer heterogeneous computing tasks. It cannot map in real time the highly differentiated characteristics of upper-layer large model inference task flows to the physical compression of the underlying control path, and it is difficult to cope with the risk of global system performance overdraft under sudden high load conditions of computing card groups.

[0005] Third, network routing and forwarding technologies based on reinforcement learning and latency estimation; for example, Chinese invention patent CN113938415B discloses a network routing and forwarding method and system based on link state estimation. It inputs the traffic matrix into the DDPG neural network model to output the first weight of each link, monitors the end-to-end latency of the network and estimates the congestion probability of each link to generate action noise, and truncates and processes the weights after adding action noise to generate the current routing strategy. Its drawback is that existing routing optimization models of this kind usually regard network layer throughput or latency as relatively isolated decision units, ignoring the memory fragmentation rate and host interruption throughput bottleneck of backend computing nodes in long time sequences; when performing cross-layer global optimization, if there is a lack of a mandatory constraint mechanism for the physical carrying capacity limit of the computing cluster, the predicted trajectory is prone to cumulative drift, causing the output routing scheme to lose its value due to exceeding the node throughput limit.

[0006] Therefore, the technical problem to be solved by this invention is how to construct a scheduling architecture with cross-layer transient state awareness and full-link collaborative optimization capabilities, so that computing card resource allocation, host interruption overhead and global network routing strategy converge to the real physical security boundary throughout the entire life cycle. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and solve the technical problems of dynamic scheduling instability of multiple heterogeneous interfaces and long-tail response delay accumulation caused by the intertwining of active detection control flow contention and network transmission jitter interference, and to provide a multi-model dynamic scheduling method for high-concurrency dialogue requests.

[0008] To achieve the above-mentioned objectives, this invention provides a multi-model dynamic scheduling method for high-concurrency dialogue requests, comprising the following steps: Step S1: Collect timestamp sequence difference to generate time interval vector: The data stream monitoring module collects the arrival timestamp sequence of 16 consecutive downlink data blocks triggering physical network card hard interrupts at the streaming downlink response data bus in the ring buffer, and calculates the absolute difference between adjacent timestamps through first-order difference to generate streaming output time interval vector. Step S2, extract the fluctuation envelope and compare it with the reference to determine the blocking index: The scheduling unit extracts the first-order differential fluctuation envelope of the streaming output time interval vector, and inputs the first-order differential fluctuation envelope into the data difference analysis module, and compares it with the reference transmission delay line from the global standard static response gateway to determine the target model to calculate the channel blocking index; Step S3: Calculate the routing suppression coefficient and update the probability distribution of the routing matrix: The matrix arbitration unit obtains the backlog length of the current request queue of the large language model as the first intermediate index, calculates the product of the first intermediate index and the channel blocking index calculated by the target model, determines the obtained product value as the real-time routing suppression coefficient, and updates the probability distribution of the cost-aware routing matrix for each active interface according to the real-time routing suppression coefficient. Step S4, compressing the window and redirecting the data stream when traffic is overloaded: When the number of concurrent input data streams per unit time exceeds the system's safe throughput threshold, the scheduling unit compresses the sliding sampling window of the arrival timestamp sequence from 10 frames to 4 frames. When the real-time routing suppression coefficient is greater than the preset safety judgment threshold of 0.85, an addressing redirection switching signal is sent to the control flow pointer register. While keeping the current existing streaming transmission context in a transparent state and without disconnecting the link, the subsequent concurrent input data streams are shifted to the active interface with a real-time routing suppression coefficient lower than 0.3.

[0009] Step S1 of the present invention includes: the data stream monitoring module captures the physical network card hard interrupt triggered by the arrival of 16 consecutive downlink data blocks on the streaming downlink response data bus; the data stream monitoring module reads the underlying hardware master clock corresponding to the physical network card hard interrupt and writes it into the random access storage queue in the order of arrival to generate an arrival timestamp sequence; the data stream monitoring module calculates the absolute difference on the time axis for adjacent items in the arrival timestamp sequence to construct the streaming output time interval vector.

[0010] The step S2 of this invention, which determines the target model's channel blocking index, includes: the scheduling unit periodically reads the average transmission delay fed back by the global standard static response gateway within a sliding window, uses the obtained average transmission delay as the reference transmission delay line, the scheduling unit extracts the first-order differential fluctuation envelope of the streaming output time interval vector, and inputs the first-order differential fluctuation envelope into the data difference analysis module. The data difference analysis module uses a delay difference analysis algorithm to subtract the reference transmission delay line from the first-order differential fluctuation envelope to calculate the target model's channel blocking index.

[0011] Step S3 of the present invention includes: the matrix arbitration unit obtains the backlog length of the current request queue of the large language model to determine the first intermediate index; the matrix arbitration unit obtains the latency constraint parameters and billing cost unit price factor input by the upstream request end, constructs a heuristic constraint screening matrix; the matrix arbitration unit multiplies the first intermediate index with the channel blocking index calculated by the target model to obtain the real-time routing suppression coefficient; the matrix arbitration unit uses the real-time routing suppression coefficient as the attenuation factor of the probability distribution and reconstructs the cost-aware routing matrix for each active interface.

[0012] In step S4 of the present invention, compressing the sliding sampling window of the arriving timestamp sequence from 10 frames to 4 frames includes: the scheduling unit monitors the network throughput in real time, compares the number of concurrent input data streams per unit time with the system's safe throughput threshold, and when the number of concurrent input data streams exceeds the system's safe throughput threshold, the scheduling unit issues a window compression command to the data stream monitoring module to compress the sliding sampling window of the arriving timestamp sequence from 10 frames to 4 frames.

[0013] The present invention describes sending an address redirection switching signal to the control flow pointer register when the real-time routing suppression coefficient is greater than the preset security judgment threshold of 0.85. This includes: when the real-time routing suppression coefficient of any active interface is detected to be greater than the preset security judgment threshold of 0.85, triggering the address redirection switching state, the matrix arbitration unit sends an address redirection switching signal to the control flow pointer register, keeping the currently established streaming session context in the transparent state, and redirecting the routing pointer of subsequent concurrent input data streams to the active interface with a real-time routing suppression coefficient lower than 0.3.

[0014] The method described in this invention also includes an overload circuit breaker limiting process: when the real-time routing suppression coefficient of any active interface is greater than the preset overload boundary threshold of 0.95, the scheduling unit sends a control signal to the data access layer to disconnect the injection of new request streams.

[0015] The method of using the obtained average transmission delay as the reference transmission delay line in step S21 of the present invention further includes: the scheduling unit periodically sends test empty packets of fixed length through a global standard static response gateway independent of the ring buffer, and calculates the average round-trip delay window value of the test empty packets on the transmission path, and uses the average round-trip delay window value as the reference transmission delay line.

[0016] The step S32 of the present invention, which adjusts the weighting parameters in the cost-aware routing matrix according to the fixed budget limit, includes: when processing the data stream, the matrix arbitration unit reads the fixed budget limit input by the upstream requesting end, adjusts the weighting parameters in the cost-aware routing matrix according to the fixed budget limit, and sends the subsequent concurrent input data streams to the corresponding activity interface.

[0017] The addressing redirection switching state described in this invention also includes the following fallback cache branch: when the scheduling unit detects that the real-time routing suppression coefficient of any active interface is greater than the preset security judgment threshold of 0.85, and there is no active interface with a real-time routing suppression coefficient lower than 0.3, the subsequent concurrent input data streams are imported into the local cache queue for queuing and caching until the real-time routing suppression coefficient is not greater than the preset security judgment threshold of 0.85.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: 1. In the multi-model dynamic scheduling of high-concurrency dialogue requests, the hardware interaction architecture of allocating independent circular buffers for each target computing port in the main control memory and maintaining write address pointers and write pointers in real time enables the system clock timer to continuously extract the absolute addressing positions of the two pointers in the physical address space within a discrete sampling period of 10μs and calculate the first intermediate parameter characterizing the relative displacement change rate of the two pointers. When the downstream computing throughput is blocked due to saturation, causing the stepping speed of the write pointer to slow down monotonically, the relative displacement change rate generates an instantaneous nonlinear transition, thereby generating a quantitative indicator reflecting the downstream computing load status in situ in the control-side memory state space. This process eliminates the traditional communication interaction action of sending probe data packets to the external network, avoids the ineffective occupation of network card interrupt resources and data bus bandwidth by probe communication, eliminates the risk of transient blocking of data flow channels caused by traditional active probe mechanisms, and ensures that the control side has an instantaneous synchronous perception of the implicit trend of underlying computing power saturation on the execution side.

[0019] 2. Based on the deduction of port saturation through the relative displacement change rate of memory address pointers, this invention further integrates the differential extraction of the physical timestamp sequence of 16 consecutive data blocks triggering hard interrupts on the streaming downlink response data bus by the data stream monitoring module. The first-order differential fluctuation envelope of the absolute difference between adjacent timestamps is calculated by hierarchically cascaded differential logic operators to form a second intermediate parameter. This parameter is then input into a preset differential filter and compared with the timing baseline of the transmission delay sliding window of the general response channel for trend comparison and noise reduction. The calculated channel blocking parameter after eliminating network transmission jitter interference is multiplied and interlocked with the relative displacement change rate shown by the first intermediate parameter. This establishes a white-box control link at the control decision layer that accurately separates physical network packet loss noise from the underlying interface calculation failure. This enables the system to accurately identify the real computing bottleneck in complex high-disturbance communication network environments, avoids cascading errors caused by occasional network jitter leading to a large-scale freezing of normally allocated ports at the routing decision layer, and maintains the stability of the routing control matrix in non-ideal communication environments.

[0020] 3. This invention utilizes a discrete compression update mechanism for the sampling time window under conditions of sudden traffic surges, along with dynamic cascading of memory bus switching instructions. When the rate of change of the upstream concurrent input data stream arrival rate monotonically increases, the system automatically compresses the sliding sampling window of the timestamp sequence from 10 frames to 4 frames to accelerate the capture of computing power overload signals. When the real-time routing suppression parameter generated by the product interlock crosses the safety judgment threshold of 0.85, the matrix arbitration unit directly sends an address redirection switching signal to the control flow pointer register. Under the premise of maintaining the current existing streaming transmission context in a transparent manner and without link interruption, the subsequent arriving concurrent input data streams are seamlessly shifted to a low-damped idle physical interface with a routing suppression parameter below 0.3. This process replaces the complex continuous optimization function solution with a hardware clock-driven discrete control flow reconstruction. While increasing the overall concurrent throughput of the system by more than 35%, it reduces the threshold computing power consumption of boundary control, enabling the system to have the dynamic adaptive fault-tolerant adjustment capability to trigger flow control backpressure and path reconstruction when a sudden computing peak touches the boundary constraints. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the multi-model dynamic scheduling process that integrates delay differential and route suppression in this invention; Figure 2 This is a schematic diagram of the hardware and software coupling architecture of the register group and memory space of this invention. Detailed Implementation

[0022] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings.

[0023] Example 1 This embodiment discloses a multi-model dynamic scheduling method for high-concurrency dialogue requests, including: Step S1: Collect timestamp sequence difference to generate time interval vector: The data stream monitoring module collects the arrival timestamp sequence of 16 consecutive downlink data blocks triggering physical network card hard interrupts at the streaming downlink response data bus in the ring buffer, and calculates the absolute difference between adjacent timestamps through first-order difference to generate streaming output time interval vector. Step S2, extract the fluctuation envelope and compare it with the reference to determine the blocking index: The scheduling unit extracts the first-order differential fluctuation envelope of the streaming output time interval vector, and inputs the first-order differential fluctuation envelope into the data difference analysis module, and compares it with the reference transmission delay line from the global standard static response gateway to determine the target model to calculate the channel blocking index; Step S3: Calculate the routing suppression coefficient and update the probability distribution of the routing matrix: The matrix arbitration unit obtains the backlog length of the current request queue of the large language model as the first intermediate index, calculates the product of the first intermediate index and the channel blocking index calculated by the target model, determines the obtained product value as the real-time routing suppression coefficient, and updates the probability distribution of the cost-aware routing matrix for each active interface according to the real-time routing suppression coefficient. Step S4, compressing the window and redirecting the data stream when traffic is overloaded: When the number of concurrent input data streams per unit time exceeds the system's safe throughput threshold, the scheduling unit compresses the sliding sampling window of the arrival timestamp sequence from 10 frames to 4 frames. When the real-time routing suppression coefficient is greater than the preset safety judgment threshold of 0.85, an addressing redirection switching signal is sent to the control flow pointer register. While keeping the current existing streaming transmission context in a transparent state and without disconnecting the link, the subsequent concurrent input data streams are shifted to the active interface with a real-time routing suppression coefficient lower than 0.3.

[0024] Step S1 in this embodiment includes: Step S11, the data stream monitoring module captures the physical network card hard interrupt triggered by the arrival of 16 consecutive downlink data blocks on the streaming downlink response data bus; Step S12, the data stream monitoring module reads the underlying hardware master clock corresponding to the physical network card hard interrupt and writes it into the random access memory queue in the order of arrival to generate an arrival timestamp sequence; Step S13, the data stream monitoring module calculates the absolute difference on the time axis for adjacent items in the arrival timestamp sequence to construct a streaming output time interval vector.

[0025] In this embodiment, step S2, determining the target model's channel blocking index, includes: Step S21, the scheduling unit periodically reads the average transmission delay fed back by the global standard static response gateway within a sliding window, and uses the obtained average transmission delay as the reference transmission delay line; Step S22, the scheduling unit extracts the first-order differential fluctuation envelope of the streaming output time interval vector and inputs the first-order differential fluctuation envelope into the data difference analysis module; Step S23, the data difference analysis module subtracts the reference transmission delay line from the first-order differential fluctuation envelope using a delay difference analysis algorithm to calculate the target model's channel blocking index.

[0026] Step S3 in this embodiment includes: Step S31, the matrix arbitration unit obtains the backlog length of the current request queue of the large language model to determine the first intermediate index; Step S32, the matrix arbitration unit obtains the latency constraint parameters and billing cost unit price factor input by the upstream requester, and constructs a heuristic constraint screening matrix; Step S33, the matrix arbitration unit multiplies the first intermediate index with the channel blocking index calculated by the target model to obtain the real-time routing suppression coefficient; Step S34, the matrix arbitration unit uses the real-time routing suppression coefficient as the attenuation factor of the probability distribution and reconstructs the cost-aware routing matrix for each active interface.

[0027] The method described in this embodiment for compressing the sliding sampling window of the arriving timestamp sequence from 10 frames to 4 frames when the number of concurrent input data streams per unit time exceeds the system's safe throughput threshold includes: Step S41, the scheduling unit monitors the network throughput in real time and compares the number of concurrent input data streams per unit time with the system's safe throughput threshold; Step S42, when the number of concurrent input data streams exceeds the system's safe throughput threshold, the scheduling unit sends a window compression command to the data stream monitoring module to compress the sliding sampling window of the arriving timestamp sequence from 10 frames to 4 frames.

[0028] The method described in this embodiment for sending an address redirection switching signal to the control flow pointer register when the real-time routing suppression coefficient is greater than the preset security judgment threshold of 0.85 includes: Step S43, when the real-time routing suppression coefficient of any active interface is detected to be greater than the preset security judgment threshold of 0.85, triggering the address redirection switching state; Step S44, the matrix arbitration unit sends an address redirection switching signal to the control flow pointer register, keeping the currently established streaming session context in the transparent state, and redirecting the routing pointer of the subsequently arriving concurrent input data stream to the active interface with a real-time routing suppression coefficient lower than 0.3.

[0029] The method described in this embodiment also includes an overload circuit breaker limiting process: when the real-time routing suppression coefficient of any active interface is greater than the preset overload boundary threshold of 0.95, the scheduling unit sends a control signal to the data access layer to disconnect the injection of new request streams.

[0030] In step S21 of this embodiment, the method of using the obtained average transmission delay as the reference transmission delay line further includes: the scheduling unit periodically sends test empty packets of fixed length through a global standard static response gateway independent of the ring buffer, and calculates the average round-trip delay window value of the test empty packets on the transmission path, and uses the average round-trip delay window value as the reference transmission delay line.

[0031] In this embodiment, step S32, constructing the heuristic constraint screening matrix, includes: step S321, when processing the data stream, the matrix arbitration unit reads the fixed budget limit input by the upstream requester; step S322, the matrix arbitration unit adjusts the weighting parameters in the cost-aware routing matrix according to the fixed budget limit, and sends the subsequent concurrent input data streams to the corresponding activity interface.

[0032] The addressing redirection switching state described in this embodiment also includes the following fallback cache branch: when the scheduling unit detects that the real-time routing suppression coefficient of any active interface is greater than the preset security judgment threshold of 0.85, and there is no active interface with a real-time routing suppression coefficient lower than 0.3, the subsequent concurrent input data streams are imported into the local cache queue for queuing and caching until the real-time routing suppression coefficient is not greater than the preset security judgment threshold of 0.85.

[0033] Example 2 In this embodiment, in a scenario where a general-purpose electronic digital data processing system handles the distribution and orchestration of high-concurrency streaming dialogue requests, the main control processor faces the risk of control flow latency caused by the interplay of heterogeneous language model interface throughput saturation and network transmission jitter. When a sudden surge of large amounts of dialogue data flows into the scheduling side, the load states of different target computing ports change drastically within milliseconds. The traditional active probing architecture, which obtains transient response latency by frequently transmitting idle probe packets, will preempt the network card hard interrupt resources and memory bus bandwidth allocated in the physical bus allocation. This will cause transient blocking of the control flow within the scheduling system at the physical network card level, resulting in cascading backlog in the upstream temporary queue and accumulating into a long tail of system processing response latency, ultimately causing a decrease in the overall throughput of the entire electronic digital data processing system. To address this, the main control processor assigns a dedicated linear circular buffer in random access memory for each available heterogeneous large language model interface at the electronic digital processing layer, setting the maximum addressing depth of the buffer to a fixed value. The underlying memory controller hard-maintains a write pointer Pin for recording the write address of upstream concurrent dialogue requests and a write pointer Pout for recording the address of downstream model API interface read and captured data streams. A high-speed system clock timer continuously extracts the absolute addressing positions of the write pointer Pin and the write pointer Pout in the memory address space at a fixed discrete sampling period of 10μs. The relative slip rate difference Dt, representing the throughput deviation, is generated by calculating the relative displacement of the write pointer Pin and the write pointer Pout within the current sampling period. Specifically, the relative slip rate difference Dt satisfies the following relationship: Where Dt is the relative slip rate difference, Pin is the address of the write pointer, and Pout is the address of the write pointer. To achieve the maximum addressing depth, when data retrieval is hindered due to the saturation of the internal computing core of a downstream specific model interface, the stepping speed of the corresponding channel's write pointer Pout slows down, causing the relative slip rate difference Dt to produce a monotonically increasing nonlinear transition at the current time section, serving as the first intermediate indicator of zero communication overhead characterizing the overload of the execution side's computing power.

[0034] Meanwhile, the data stream monitoring module in the main control processor intercepts the streaming downlink response data bus of the ring buffer in real time, captures the physical network card hard interrupt triggered by the arrival of 16 consecutive downlink data blocks on the streaming downlink response data bus, and reads the underlying hardware master clock corresponding to the physical network card hard interrupt and writes it sequentially into the random access memory queue according to the arrival time sequence to generate an arrival timestamp sequence. The absolute difference on the time axis is calculated for adjacent items in the arrival timestamp sequence to construct a streaming output time interval vector. The specific character time interval In in the streaming output time interval vector satisfies the relation... Where In is the character time interval, Tn is the timestamp of the nth character currently output, and Tn-1 is the timestamp of the previous character. The first-order differential fluctuation envelope of the streaming output time interval vector is extracted using the built-in differential logic operator as a second intermediate index. The first derivative of the time interval between adjacent characters in the first-order differential fluctuation envelope... Satisfying the relation ,in, Let In be the first derivative, In be the current character time interval, and In-1 be the previous character time interval. The main control processor inputs the first-order differential fluctuation envelope into the data difference analysis module. It synchronously and periodically reads the average transmission delay fed back by the global standard static response gateway within the sliding window and uses the obtained average transmission delay as the reference transmission delay line. The reference transmission delay line is obtained by the global standard static response gateway, which is independent of the ring buffer, periodically sending fixed-length test empty packets and statistically calculating the average of the round-trip delay sliding window of the test empty packets on the transmission path. The data difference analysis module uses the delay difference analysis algorithm to subtract the reference transmission delay line from the first-order differential fluctuation envelope to calculate the target model's channel blocking index. This is to remove the false delay noise caused by transmission network jitter and to offset the risk of pointer rate misjudgment caused by random jitter of the external transmission network.

[0035] The main control processor uses a multi-core floating-point computing platform with a base clock frequency of 3.2GHz and a memory bandwidth of 51.2GB / s, and the maximum addressing depth of the circular buffer is [not specified]. Set to 65536 bytes, when the circular buffer receives data streams from concurrent external dialogue requests, the network protocol stack kernel driver performs one-way hash desensitization on sensitive fields of the input text stream, limiting feature analysis to downlink data block counting and arrival time sequence, blocking unnecessary privacy data from being transmitted downwards. Based on the principle of homogeneity of time series variance, to match the spatiotemporal scale of character arrival time jitter fluctuations and global average transmission delay, the data difference analysis module uses the first-order difference fluctuation envelope of the streaming output time interval vector. Transform into satisfying relation Normalized jitter intensity ,in, Greater than or equal to 0, The first derivative of the time interval between adjacent characters. The sliding window provides feedback on the average transmission delay, which is greater than 0. The data difference analysis module compares... By directly subtracting the background noise baseline from the preset jitter stability boundary, the dimensionless target model is output to calculate the channel blocking index.

[0036] The matrix arbitration unit obtains the backlog length of the current request queue of the large language model as the first intermediate index, calculates the product of the first intermediate index and the channel blocking index calculated by the target model, and determines the resulting product as the real-time routing suppression coefficient. Simultaneously, it obtains the latency constraint parameters, billing cost unit price factor, and fixed budget limit input from the upstream requester to construct a heuristic constraint screening matrix. Based on the fixed budget limit, it adjusts the weighting parameters in the cost-aware routing matrix and uses the real-time routing suppression coefficient as a hard attenuation factor in the probability distribution to dynamically reconstruct the probability distribution of the cost-aware routing matrix for each active interface. When the number of concurrent input data streams per unit time exceeds the system's safe throughput threshold, the scheduling unit monitors the network throughput in real time and issues a window compression command to the data stream monitoring module, hard compressing the sliding sampling window of the arrival timestamp sequence from 10 frames to 4 frames to accelerate the capture of computing power overload signals. Once the real-time routing suppression coefficient of any active interface is detected to be greater than... When the preset security threshold is 0.85, an address redirection switching state is triggered. The matrix arbitration unit directly sends an address redirection switching signal to the control flow pointer register. While maintaining the current existing streaming transmission context in a transparent state and without disconnecting the link, the routing pointers of subsequent concurrent input data streams are redirected and addressed to active interfaces with a real-time routing suppression coefficient lower than 0.3. When the scheduling unit detects that the real-time routing suppression coefficient of any active interface is greater than the preset security threshold of 0.85 and there are no active interfaces with a real-time routing suppression coefficient lower than 0.3, the subsequent concurrent input data streams are imported into the local buffer queue for queuing and buffering until the real-time routing suppression coefficient is not greater than the preset security threshold of 0.85. If the real-time routing suppression coefficient of any active interface continues to rise and exceeds the preset overload boundary threshold of 0.95, the scheduling unit sends a control signal to the data access layer to disconnect the injection of new request streams in order to forcibly limit the injection rate of new streams.

[0037] In a distributed clustered deployment, the scheduling unit uses a multiplexing distribution mechanism to distribute global concurrent tasks to multiple independent heterogeneous language model interface nodes. Within the system's operating range, the safety threshold is between 0.80 and 0.90. Concurrent throughput stress test data shows that when the safety threshold is below 0.80, the control flow pointer oscillates at high frequency between backup interface nodes, increasing the multiplexing overhead of the system memory bus. When the safety threshold is above 0.90, the system lags in triggering redirection, causing the backlog length of the temporary queue to exceed the safety redundancy. Test data determines that a safety threshold of 0.85, an active interface shift threshold of 0.30, and a preset overload boundary threshold of 0.95 result in a high-concurrency environment processing response latency of 4. With a time of 5.3ms, the system's safe throughput is improved by more than 35%. To ensure the physical and engineering scientific validity of the above parameter selection, the boundary ranges of the preset safety judgment threshold, active interface shift threshold, preset overload boundary threshold, and system-level pressure boundary threshold are determined through prior system load boundary stress testing and discrete delay variance convergence derivation. In actual engineering deployment, when the real-time routing suppression coefficient is below the lower bound of 0.30, it indicates that the channel computing power is extremely idle and bus addressing is smooth. At this time, shifting traffic can ensure no queuing delay. If the shift threshold is below 0.30, it will lead to insufficient utilization of the backup interface. When the coefficient crosses the safety upper bound of 0.85, queue backlog and hardware interrupt overhead begin to nonlinearly superimpose, triggering the table. Needle-based traffic splitting can precisely offset long-tail response latency. However, if the threshold exceeds 0.85, it can cause cascading blocking of the bus control flow pointer in the register, leading to temporary queue backlog exceeding safety redundancy and resulting in large-scale long-tail queuing and packet loss crashes. Through sensitivity testing in a simulation environment with continuous adjustments to various boundaries in 50ms increments, the threshold parameter range for the aforementioned interlocking was ultimately determined, achieving an optimal balance between routing redirection bus overhead and end-to-end response latency control. The system is configured with a bidirectional adaptive adjustment mechanism. When the number of concurrent input data streams falls below the system's safe throughput threshold and the real-time routing suppression coefficient of the active interface drops below 0.30, the scheduling unit initiates uplink recovery adjustment, sending data to the control flow pointer register. Sending a reverse addressing redirection switching signal, the sliding sampling window is restored from 4 frames to 10 frames, improving the smoothness of latency under stable operating conditions. In overload conditions where the real-time routing suppression coefficient of all active interfaces is greater than 0.85 and the local cache queue reaches its full load limit, the scheduling unit calls the built-in lightweight local streaming degradation interception algorithm, stops calculating the cost-aware routing matrix, directly truncates the newly injected concurrent data stream and converts it into a fixed-format overload prompt text block to return to the requesting end. The real-time routing suppression coefficient is directly changed by the hardware driver to change the addressing bias of the control flow pointer register, limiting the jump range of the control flow pointer, driving the subsequent arrival of the dialogue request stream into the low-damped physical interface, realizing the direct constraint of the control algorithm on the underlying computational routing actions.

[0038] Example 3 In this embodiment, when the system runs in a distributed multilingual model scheduling verification simulation environment, the test platform is based on an interconnected computing architecture with electrical digital data processing capabilities. The main control processor has multi-core parallel floating-point operation capabilities, a base clock frequency set to 3.2GHz, and a random access memory (RAM) including a high-speed dual-channel bus architecture with a bandwidth of no less than 51.2GB / s to simulate a streaming data environment with a throughput of no less than 50,000 concurrent dialogue requests per second. Randomly distributed, disordered network transmission jitter is introduced as interference noise into the original request stream of the test signal source, and a non-Gaussian delay disturbance with a mean square error of 15ms is superimposed in the downlink communication link to establish a test benchmark for a non-ideal engineering environment. In the working state settings of the linear circular buffer, the maximum addressing depth is... The numerical setting of the ring buffer balances the constraint between the physical coverage period of the ring buffer and the risk of memory addressing overflow of the main control processor, and its value is determined to be 65536 bytes. The sampling period setting of the high-speed system clock timer is constrained by both system bus interrupt overhead and transient exception capture sensitivity. By limiting the sampling period to a fixed discrete interval of 10μs, pointer sliding resolution is provided without triggering processor hard interrupt overload. This constructs a closed and reproducible initial boundary environment for parameter optimization of the entire scheduling architecture. In the experiment verifying the distributed multi-language model dynamic scheduling method, the experimental setup includes an experimental group using complete technical means, and a group lacking memory access. The experimental group, the first control group, lacked physical network card hard interrupt arrival timestamp differential calculation; the third control group had its security judgment threshold lowered to 0.70; and the fourth control group had its security judgment threshold raised to 0.98. When concurrent dialog request sources continuously wrote to the linear circular buffer at an initial flow rate of 8000 characters per second, the step displacements of the write pointer Pin and the write pointer Pout collected in the experimental group, the first control group, and the second control group remained synchronized under the initial non-overload state. The corresponding dimensionless relative slip rate difference Dt was maintained within the range of 0.12 to 0.15. The specific relative slip rate difference Dt satisfies the following relationship: Where Dt is the relative slip rate difference, which is a dimensionless pure number; Pin is the memory address of the write pointer that records the write address of the upstream concurrent dialogue request; and Pout is the memory address of the write pointer that records the write address of the downstream model interface reading and fetching data stream. This represents the maximum addressing depth of the linear circular buffer.

[0039] As the concurrent dialogue request rate fluctuated to 45,000 characters per second within 50ms, the downstream model interface experienced data capture stagnation due to saturated processing power. The write pointer Pin of each group experienced address advance, and the write pointer Pout experienced step lag. The experimental group used a relational formula to calculate the relative slip rate difference Dt in real time, which showed an increasing trend, rising from 0.14 to 0.86 within 12.5ms, exceeding the safety threshold of 0.85. Simultaneously, its data stream monitoring module intercepted the streaming downlink response data bus, collecting the arrival timestamp sequence of 16 consecutive downlink data blocks triggering physical network card hard interrupts. The absolute difference between adjacent timestamps was calculated using first-order difference to generate a streaming output time interval vector. The character time interval In in the specific streaming output time interval vector satisfies the relational formula. Where In is the character time interval, Tn is the arrival timestamp of the nth character in the current output, and Tn-1 is the arrival timestamp of the previous character. The indices n and n-1 are the sample indices of the time series. Furthermore, the built-in differential logic operator is used to extract the first-order difference fluctuation envelope of the streaming output time interval vector. The first derivative of the time interval between adjacent characters in the first-order difference fluctuation envelope is... Satisfying the relation ,in, Let In be the first derivative of the time interval between adjacent characters, In be the current character time interval, In-1 be the previous character time interval, and the subscript n be the sample index of the time series. The main control processor inputs the first-order differential fluctuation envelope into the data difference analysis module and compares it with the reference transmission delay line from the global standard static response gateway. The reference transmission delay line is deducted from the first-order differential fluctuation envelope, and the target model after excluding network jitter is separated under the current working condition to calculate the channel blocking index, so that the real-time routing suppression coefficient is kept at a value of 0.88, forming a quantitative extraction of the underlying interface computing power blocking state.

[0040] When the real-time routing suppression coefficient exceeded the security threshold of 0.85, the scheduling unit of the test group responded to the window compression command and compressed the sliding sampling window to 4 frames. After receiving the address redirection switching signal, the control flow pointer register redirected the routing pointers of subsequent concurrent data streams to the active interface with a real-time routing suppression coefficient of 0.25. This reduced the backlog length of the upstream temporary queue from 42 requests in an overloaded state to 5 requests, and the overall system processing response latency remained within 45.3ms. In contrast, the first control group, lacking memory monitoring, could not calculate the relative slip rate difference Dt, causing the routing control matrix to fail to detect millisecond-level computing power mutations. This resulted in cascading backlogs in the upstream temporary queue and a processing response latency of 358.6ms. The second control group, lacking interrupt timestamp calculation, failed to remove network transmission jitter. The first control group, with a security threshold of 0.70, exhibited a non-linear numerical response. The low threshold caused frequent erroneous triggering of address redirection states, resulting in the control flow pointer switching between interfaces, leading to address register rewriting latency and memory bus reuse overhead. This resulted in an overall decrease in concurrent throughput of 18.4% compared to the experimental group. The second control group, with a security threshold of 0.98, entered the computing power saturation zone. Due to the excessively high trigger condition, the system delayed address redirection until the queue backlog reached 245 requests. At this point, the processing response latency showed a monotonically increasing trend. All test data indicate that a security threshold of 0.85 is the working window boundary for balancing routing redirection overhead and response latency control.

[0041] Example 4 In this embodiment, during streaming request processing in a general-purpose industrial computing node, the main control processor interacts with the interface processing module containing heterogeneous model interfaces via a memory address bus. When the concurrent request rate for a specific dialogue task transiently spikes to over 40,000 requests per second, the model API interface located in the execution layer experiences localized computing power fluctuations due to frequent invalidations of its internal key-value cache addresses. These fluctuations are instantaneously transmitted to the system bus via the memory controller, manifesting as a transient jump in the memory circular queue write pointer Pout. The real-time routing decision module in the main control processor obtains the relative slip rate difference Dt of the corresponding interface through the data flow monitoring module and performs a nonlinear mapping of the relative slip rate difference Dt based on a preset computing power overload response mapping model. The specific computing power overload response mapping model satisfies the following relation... Where Rs is the computing power overload response mapping value, k is the preset linear adjustment coefficient, and Dt is the relative slip rate difference. To eliminate the apparent dimensional mismatch caused by cross-system level data interaction, the linear adjustment coefficient... The physical units are defined using a cross-dimensional bridging normalization rule in practical operations; due to the difference in relative slip rates... This reflects the absolute address offset deviation ratio of the circular buffer in physical memory space. At the code implementation level, it essentially carries a hidden dimension of the buffer addressing depth in bytes. Therefore, a linear adjustment coefficient... The value is specified as the maximum addressing depth of the circular buffer. It is proportional to the reciprocal of the value. When the byte component in its unit is multiplied with the underlying pointer absolute address space, dimensional offsetting and cancellation are performed through the digital-to-physical conversion interface, thereby ensuring that the final output computing power overload response mapping value is accurate. At the logical level, it reverts to a completely dimensionless deterministic scalar pure number, ensuring the absolute self-consistency of the formula in terms of mathematical model and physical dimensions. Through this mapping relationship, the system amplifies the small change in the relative slip rate difference Dt into the fluctuation of the computing power overload response mapping value Rs, thereby predicting the control flow distribution direction before computing power saturation is triggered. Once the computing power overload response mapping value Rs is detected to exceed the set system-level pressure boundary threshold of 0.92, the main control processor directly sends an address redirection switching signal to the control flow pointer register, redirecting the subsequent request flow to the backup interface where the computing power overload response mapping value Rs is lower than 0.35. This avoids the long-tail queuing phenomenon caused by the exhaustion of computing power of a single model interface, ensuring the stability of the response latency of the dialogue system under high concurrency.

[0042] Example 5 In this embodiment, during the initial deployment of a distributed multilingual model scheduling and verification simulation environment, the main control processor uses the baseline self-test module in the bootloader register to traverse the system's addressable random access memory space to calculate the upper limit capacity of available memory blocks, and assigns the maximum addressing depth of the linear circular buffer for each model interface according to the allocation rules. Simultaneously, the benchmark self-test module continuously sends 50 fixed-length test empty packets to the global standard static response gateway to statistically analyze their round-trip delay. By performing a sliding window mean operation on the collected delay sequence, it locks the basic data reflecting the background network topology of the site and establishes a benchmark transmission delay line as the background noise baseline. After the benchmark transmission delay line is established, the main control processor uses a micro-calibration logic operator to perform linear regression fitting on the bus hard interrupt frequency and distribution delay hysteresis data under historical overload conditions to calibrate the parameter error. The system bus overflow critical point value obtained from the test set analysis is written into a non-volatile register to solidify the preset safety judgment threshold. The adaptive window compression mechanism and multi-object orthogonal routing translation logic obtain a decision control surface aligned with the current physical hardware environment specifications. When subsequent concurrent dialogue request flows in, the scheduling unit outputs a routing switching control flow according to the above-mentioned field calibration parameters. The entire electronic digital data processing system is in a hardware and software coupled working state.

[0043] Example 6 In this embodiment, within a standard static response gateway, due to random jitter in the external transmission network, the data stream arrives at the large language model interface accompanied by nondeterministic network jitter noise. To accurately identify transient congestion of the internal computing channel caused by internal key-value cache saturation or streaming output buffer contention, the system employs a pre-deployment calibration method. When the system is deployed in a specific high-concurrency communication network environment, the processor executes a pre-calibration procedure to establish an environmental background noise baseline. The main control processor calls the feature parameter extraction module to collect the computing power utilization rate of the computing channel within a unit processing cycle, using this as an input parameter and substituting it into a preset feature mapping formula to obtain the transient congestion feature value of the current channel. The calculation satisfies the relation ,in, This is a transient blocking characteristic value. To calculate the computing power utilization rate of the channel within a unit processing cycle, To calculate the number of backlogged requests in the current input queue of the channel, To calculate the average processing time per request for the channel, the scheduling unit determines the transient blocking characteristic value. The mean value over three consecutive sampling periods exceeds the preset congestion criterion threshold. When a routing logic reconfiguration instruction is triggered, the scheduling unit reads the heuristic load balancing weight table stored in the non-volatile register, forcibly reduces the routing weight of the currently saturated computing channel to below 0.05, and shifts the incoming request flow to the transient blocking characteristic value. In candidate computing channels with a latency of less than 0.20, the queuing lag time of the overall system control flow is reduced and the risk of cascading response failure caused by load imbalance is eliminated.

Claims

1. A multi-model dynamic scheduling method for high-concurrency dialogue requests, characterized in that, include: Step S1: The data stream monitoring module collects the arrival timestamp sequence of 16 consecutive downlink data blocks triggering physical network card hard interrupts at the streaming downlink response data bus in the ring buffer, and calculates the absolute difference between adjacent timestamps through first-order difference to generate a streaming output time interval vector. In step S2, the scheduling unit periodically reads the average transmission delay fed back by the global standard static response gateway within the sliding window and uses the obtained average transmission delay as the reference transmission delay line. The scheduling unit extracts the first-order differential fluctuation envelope of the streaming output time interval vector and inputs the first-order differential fluctuation envelope into the data difference analysis module. The data difference analysis module uses the delay difference analysis algorithm to subtract the reference transmission delay line from the first-order differential fluctuation envelope to calculate the target model's channel blocking index. Step S3: The matrix arbitration unit obtains the backlog length of the current request queue of the large language model to determine the first intermediate index; The matrix arbitration unit obtains the delay constraint parameters and billing cost unit price factor input by the upstream requester and constructs a heuristic constraint screening matrix. The matrix arbitration unit multiplies the first intermediate index with the channel blocking index calculated by the target model to obtain the real-time routing suppression coefficient. The matrix arbitration unit uses the real-time routing suppression coefficient as the decay factor of the probability distribution and combines it with the heuristic constraint screening matrix to reconstruct the cost-aware routing matrix for each activity interface, so as to update the probability distribution of the cost-aware routing matrix for each activity interface. Step S4: When the number of concurrent input data streams per unit time exceeds the system security throughput threshold, the scheduling unit compresses the sliding sampling window of the arrival timestamp sequence from 10 frames to 4 frames. When the real-time routing suppression coefficient is greater than the preset security judgment threshold of 0.85, it sends an addressing redirection switching signal to the control flow pointer register. While keeping the current existing streaming transmission context in a transparent state and without disconnecting the link, the subsequent concurrent input data streams are shifted to the active interface with a real-time routing suppression coefficient of less than 0.

3.

2. The multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 1, characterized in that, Step S1 includes: Step S11, the data stream monitoring module captures the physical network card hard interrupt triggered by the arrival of 16 consecutive downlink data blocks on the streaming downlink response data bus; Step S12, the data stream monitoring module reads the underlying hardware master clock corresponding to the physical network card hard interrupt and writes it into the random access memory queue in the order of arrival to generate an arrival timestamp sequence; Step S13, the data stream monitoring module calculates the absolute difference on the time axis for adjacent items in the arrival timestamp sequence to construct a streaming output time interval vector.

3. The multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 1, characterized in that, Step S4, which compresses the sliding sampling window of the arriving timestamp sequence from 10 frames to 4 frames, includes: Step S41, the scheduling unit monitors the network throughput in real time and compares the number of concurrent input data streams per unit time with the system's safe throughput threshold; Step S42, when the number of concurrent input data streams exceeds the system's safe throughput threshold, the scheduling unit sends a window compression command to the data stream monitoring module to compress the sliding sampling window of the arriving timestamp sequence from 10 frames to 4 frames.

4. The multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 1, characterized in that, When the real-time routing suppression coefficient is greater than the preset security threshold of 0.85, sending an address redirection switching signal to the control flow pointer register includes: Step S43, when the real-time routing suppression coefficient of any active interface is detected to be greater than the preset security threshold of 0.85, triggering the address redirection switching state; Step S44, the matrix arbitration unit sends an address redirection switching signal to the control flow pointer register, keeping the currently established streaming session context in the transparent state, and redirecting the routing pointer of the subsequently arriving concurrent input data stream to the active interface with a real-time routing suppression coefficient lower than 0.

3.

5. The multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 1, characterized in that, The method also includes an overload circuit breaker limiting process: when the real-time routing suppression coefficient of any active interface is greater than the preset overload boundary threshold of 0.95, the scheduling unit sends a control signal to the data access layer to disconnect the injection of new request streams.

6. The multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 1, characterized in that, The method of using the obtained average transmission delay as the reference transmission delay line in step S2 includes: the scheduling unit periodically sends test empty packets of fixed length through a global standard static response gateway independent of the ring buffer, and calculates the average round-trip delay window value of the test empty packets on the transmission path, and uses the average round-trip delay window value as the reference transmission delay line.

7. A multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 1, characterized in that, The construction of the heuristic constraint filtering matrix in step S3 includes: when processing the data stream, the matrix arbitration unit reads the fixed budget limit input from the upstream requester; the matrix arbitration unit adjusts the weighting parameters in the cost-aware routing matrix according to the fixed budget limit, and sends the subsequent concurrent input data streams to the corresponding activity interfaces.

8. A multi-model dynamic scheduling method for high-concurrency dialogue requests according to claim 4, characterized in that, The address redirection switching state also includes the following fallback cache branch: when the scheduling unit detects that the real-time routing suppression coefficient of any active interface is greater than the preset security judgment threshold of 0.85, and there is no active interface with a real-time routing suppression coefficient lower than 0.3, the subsequent concurrent input data streams are imported into the local cache queue for queuing and caching until the real-time routing suppression coefficient is not greater than the preset security judgment threshold of 0.85.

Images