A multi-tenant large model architecture-oriented call link exception monitoring method

Abstract

The application relates to the technical field of large model distributed link monitoring, in particular to a call link abnormality monitoring method for a multi-tenant large model architecture, which comprises the following steps: a gateway extracts a tracking context of an inference request and transmits the tracking context to a working node; the node obtains actual time consumption, floating point operation amount and display memory flow, analyzes a radix tree to obtain cache block topology coordinates and calculates a decay penalty factor; a dynamic bandwidth boundary is obtained by combining the penalty factor and a theoretical bandwidth, a hardware performance boundary model is input, and a theoretical expected delay baseline is output; a time consumption residual sequence is calculated, a high-pass sub-band energy mean value is extracted as a physical fragmentation index through discrete wavelet transform; when the index is greater than a threshold value, it is determined that a bottom layer memory access abnormality occurs, and an alarm log is generated, so that the display memory is regularly arranged, and candidate blocks are asynchronously copied to continuous allocator page groups in the same topology domain according to priorities. The application realizes link abnormality elimination through display memory asynchronous arrangement.

Description

Technical Field

[0001] This invention relates to the field of large-scale distributed link monitoring technology, specifically a method for monitoring call link anomalies in multi-tenant large-scale model architectures. Background Technology

[0002] With the rapid development of artificial intelligence technology, distributed large-model inference architectures shared by multiple tenants are widely used. To improve the concurrent throughput of the cluster, the system usually adopts continuous batch processing technology and dynamic key-value cache (KV Cache) allocation mechanism. In multi-tenant scenarios, due to the high differences in the length of prompt words, the number of concurrent requests, and the length of generated text among different tenants, as inference requests are constantly executed and completed, the global memory pool will naturally generate a large amount of memory space fragmentation during the frequent dynamic allocation and release process.

[0003] During the autoregressive decoding phase of large models, system performance places extremely high demands on the efficiency of memory access communication at the underlying GPU memory. When memory fragmentation accumulates to a certain extent, the discrete distribution of cache blocks in physical space leads to a significant increase in cross-domain or cross-node communication costs. This memory access lag at the underlying hardware level will directly propagate upwards to the application layer, manifesting as a spike in latency in multi-tenant call chains (such as first-word response timeouts, sudden increases in single-word generation time, etc.), which in turn can trigger service level agreement (SLA) breaches.

[0004] In routine system operation and resource management, global memory defragmentation mechanisms are typically used to address memory fragmentation issues and restore system performance. However, conventional defragmentation operations usually require suspending or halting the currently executing computation stream (i.e., triggering a Stop-the-world mechanism) to ensure memory access consistency and security when moving data and redirecting pointers. This approach can cause a momentary halt to the online inference service for all tenants within the cluster, failing to meet the stringent service continuity requirements of high-concurrency, low-latency inference applications.

[0005] In summary, how to accurately monitor memory access latency anomalies in the call chain caused by physical memory fragmentation and dynamically eliminate such anomalies is a technical problem that urgently needs to be solved in this field.

[0006] To address this, a method for monitoring call chain anomalies in a multi-tenant large-scale architecture is proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a call link anomaly monitoring method for multi-tenant large-model architectures, which achieves link anomaly elimination through asynchronous memory regularization. The method includes: the gateway extracting the tracing context of inference requests and passing it through to worker nodes; the nodes obtaining actual latency, floating-point operation volume, and memory traffic, parsing the radix tree to obtain the cache block topology coordinates and calculating the attenuation penalty factor; combining the penalty factor with theoretical bandwidth to obtain the dynamic bandwidth boundary, inputting it into the hardware performance boundary model to output the theoretical expected latency baseline; calculating the latency residual sequence, extracting the high-pass subband energy mean as a physical fragmentation index through discrete wavelet transform; when the index exceeds a threshold, determining that a low-level memory access anomaly has occurred and generating an alarm log, thereby triggering memory regularization, and asynchronously copying candidate blocks to contiguous allocator page groups within the same topology domain according to priority.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for monitoring call chain anomalies in a multi-tenant large-scale architecture includes: The main scheduling gateway extracts the tracing context of the inference request and passes it through to the target worker node; The hardware micro-events of the target working node are collected to obtain the actual observation time, physical floating-point operation volume and video memory traffic. The radix tree is parsed to obtain the topological level coordinates of the hit cache block, and then substituted into the preset topology-aware spatial locality decay matrix to calculate the decay penalty factor. The theoretical peak memory bandwidth is calculated by using a decay penalty factor to obtain the topological dynamic memory bandwidth boundary; the physical floating-point operation value, memory flow value and the topological dynamic memory bandwidth boundary are input into the hardware performance boundary model to output the theoretical expected latency baseline. The difference between the actual observation time and the theoretical expected delay baseline is calculated to generate a time-domain delay residual sequence; a discrete wavelet transform is performed on the time-domain delay residual sequence to extract the mean energy of the high-pass subband, which is used as a physical fragmentation index. When the physical fragmentation index exceeds the preset fragmentation threshold, it is determined that a low-level memory access anomaly has occurred in the call chain, an alarm log is generated, and memory consolidation is triggered. Hit cache blocks with non-contiguous logical addresses are extracted as candidate blocks. The sorting priority is calculated based on the communication cost corresponding to the topology level coordinates of the candidate blocks. The candidate blocks are asynchronously copied to the contiguous allocator page group in the same topology domain according to the sorting priority.

[0009] Preferably, the process of extracting the tracing context of the inference request through the main scheduling gateway and transmitting it to the target worker node includes: intercepting the inference request containing text data input by the current tenant at the main scheduling gateway; extracting the tenant identifier and a preset service level agreement (SLA) threshold from the inference request, wherein the SLA threshold includes a first-character response time threshold and a single-character generation time threshold; combining the tenant identifier and the SLA threshold to generate the tracing context; establishing a cross-process shared memory channel between the main scheduling gateway and the target worker node; and using the cross-process shared memory channel to map and transmit the tracing context to the isolated operating environment within the target worker node.

[0010] Preferably, the process of obtaining the topological level coordinates of the hit cache block by parsing the radix tree includes: starting a non-intrusive probe in the isolated operating environment of the target worker node; dynamically loading the corresponding underlying hardware performance analysis library according to the current hardware chip type of the target worker node; collecting hardware micro-events by listening to the underlying hardware registers through the underlying hardware performance analysis library, and extracting the actual observation time, physical floating-point operation volume and video memory traffic values; and during the initialization phase of the distributed inference system, constructing a static mapping from the logical video memory block to the heterogeneous interconnect architecture topology level using the combination key of the device number and the logical video memory block identifier as an index. The mapping table includes on-chip high-speed interconnect domain, cross-chip switching domain, and cross-host interconnect domain. It periodically collects measured communication delay samples between adjacent working nodes. When the deviation of the measured communication delay sample from the calibration value exceeds a preset drift threshold, it corrects the bandwidth reference value in the static mapping table and generates a dynamic calibration mapping table. When the inference request enters the decoding stage, it parses the cache management status of the large model inference engine in the radix tree and extracts the allocation and release instructions of the dynamic key-value cache block. It obtains the combination key corresponding to the currently hit cache block, queries the dynamic calibration mapping table with the combination key, and extracts the corresponding topology level coordinates.

[0011] Preferably, the process of calculating the attenuation penalty factor includes: reconstructing the memory allocation state using allocation and release instructions, aligning the reconstructed memory allocation state with topology level coordinates, and extracting the distribution data of hit key-value cache blocks and their associated cache blocks with topology level coordinate attributes; using the hit key-value cache blocks and their associated cache blocks with parent-child or prefix sharing relationships in the radix tree as matrix nodes, mapping the topology level coordinates of each matrix node to a preset topology-aware spatial locality attenuation matrix to generate initial matrix elements; comparing the topology level coordinates corresponding to the parent and child nodes that generate logical branches in the radix tree to determine whether the logical branch crosses a hardware physical boundary with different communication bandwidth levels; when it is determined that the hardware physical boundary is crossed, extracting the current effective bandwidth ratio at both ends of the hardware physical boundary from the dynamic calibration mapping table; constructing an exponential penalty function with the current effective bandwidth ratio as an independent variable, using the exponential penalty function to perform weight scaling calculation on the initial matrix elements corresponding to the logical branch in the topology-aware spatial locality attenuation matrix, outputting a nonlinear bandwidth attenuation penalty factor, and using the nonlinear bandwidth attenuation penalty factor as the attenuation penalty factor.

[0012] Preferably, the process of outputting the theoretical expected latency baseline includes: extracting the factory-preset theoretical peak memory bandwidth of the underlying hardware chip in the target working node, and multiplying the theoretical peak memory bandwidth by the attenuation penalty factor to calculate the topology dynamic memory bandwidth boundary; extracting the theoretical peak computing power value of the underlying hardware chip in the target working node, and extracting the operator effective utilization rate conversion factor pre-calibrated through offline benchmark testing, multiplying the theoretical peak computing power value by the operator effective utilization rate conversion factor to calculate the empirical effective computing power peak; and combining the physical floating-point operation value, memory traffic value, empirical effective computing power peak value, and... The topology dynamic memory bandwidth boundary is input into a preset hardware performance boundary model; in the hardware performance boundary model, a first ratio of the physical floating-point operation quantity to the empirical effective computing power peak value is calculated, and the first ratio is used as the computing power-limited theoretical latency; the memory traffic quantity is calculated to a second ratio of the topology dynamic memory bandwidth boundary, and the second ratio is used as the memory access-limited theoretical latency; the maximum value between the computing power-limited theoretical latency and the memory access-limited theoretical latency is extracted to generate a theoretical expected latency baseline with fragmentation-limited characteristics, and the theoretical expected latency baseline with fragmentation-limited characteristics is used as the theoretical expected latency baseline.

[0013] Preferably, the process of using physical fragmentation as an indicator includes: calculating the difference between the actual observation time corresponding to a single decoding and the theoretically expected delay baseline; arranging and concatenating multiple differences within a preset time window in chronological order to generate the time-domain delay residual sequence; constructing a wavelet decomposition filter bank containing a high-pass filter using preset wavelet basis functions; inputting the time-domain delay residual sequence into the wavelet decomposition filter bank for multi-scale discrete decomposition, and extracting the high-pass subband coefficient sequence output by the high-pass filter; and extracting the discrete coefficients of each item in the high-pass subband coefficient sequence. The system calculates the square of each discrete coefficient to obtain the corresponding single-point energy value; it then calculates the arithmetic mean of all single-point energy values ​​within the high-pass subband coefficient sequence to generate the high-pass subband energy mean; it simultaneously collects orthogonal hardware interference micro-events including device temperature downclocking flags, bus retransmission rate, and thread context switching rate, and generates an exclusive verification mask based on a preset interference judgment logic; it then uses the exclusive verification mask to perform multi-source interference filtering on the high-pass subband energy mean to obtain a corrected quantization value characterizing the degree of pure physical memory access jitter, and uses the corrected quantization value as the physical fragmentation index.

[0014] Preferably, the process of asynchronously copying candidate blocks to contiguous allocator page groups within the same topology domain according to sorting priority includes: monitoring the running phase of the large model inference engine in the target worker node; generating an alarm log when the physical fragmentation index exceeds a preset threshold, and determining that the large model inference engine is in a distributed inference communication bubble period, causing the underlying memory bus to be in a low-load idle state, triggering memory regularization; traversing the radix tree, filtering out active cache blocks with internal reference counts greater than zero and non-contiguous physical storage addresses, and using the active cache blocks as candidate blocks; extracting the topology level coordinates of each candidate block, and extracting the corresponding topology level from the dynamic calibration mapping table. The cross-domain communication delay calibration value of the coordinates is used; the cross-domain communication delay calibration value is forward mapped to calculate the merging benefit score of each candidate block, and the merging benefit score is used as the sorting priority. A copy execution queue is generated from high to low according to the sorting priority; a background direct memory access transport stream independent of the main computing stream is established in the target working node; using the background direct memory access transport stream, the candidate blocks ranked first are asynchronously copied to the reserved contiguous allocator page group in the same topology domain along the copy execution queue; the addressing pointers of the corresponding copied candidate blocks in the radix tree are redirected to the contiguous addressable logical pages in the contiguous allocator page group in the same topology domain.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By comparing the logical branches of the radix tree with the underlying hardware physical boundaries, and introducing an exponential penalty function based on the current effective bandwidth ratio, during large-scale memory access, cache blocks scattered in low-speed interconnect domains can cause nonlinear communication congestion. This transforms the cost of cross-domain communication into a numerical attenuation penalty factor, performing weight scaling calculations on the elements of the topology-aware matrix. This design establishes a mathematical mapping between the physically dispersed state and the underlying bandwidth loss, enabling the system to accurately calculate the dynamic memory bandwidth boundaries under constrained conditions, providing a physically realistic computational basis for multi-tenant baseline generation.

[0016] 2. By performing multi-scale discrete decomposition on the time-domain delay residual sequence, and extracting the discrete coefficients of the high-pass filter output to calculate the energy mean, the difference sequence is obtained word by word. Wavelet decomposition is used to filter out low-frequency trends, and the high-frequency delay spikes caused by memory fragmentation are squared. This signal separation mechanism decomposes the complex system operation time and outputs a single quantitative value that represents the degree of physical memory access jitter, thus eliminating the physical interference of computing load changes on the monitoring results.

[0017] 3. By monitoring the inference engine during the distributed inference communication bubble period and triggering micro-regulation instructions, and establishing an independent background direct memory access transmission stream, the underlying video memory bus is utilized during the low-load physical gap. The merging benefit is calculated based on the cross-domain communication cost, and asynchronous copying is performed on candidate blocks according to priority. This hardware and software collaborative scheduling method prioritizes the physical address contiguous reorganization and pointer redirection of scattered video memory with the highest communication cost without intruding on the main computing flow, thus converging the spatial fragmentation state. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a method for monitoring call chain anomalies in a multi-tenant large-scale model architecture, provided by this invention. Figure 2 A schematic diagram of the calculation process for the attenuation penalty factor provided by this invention; Figure 3 This is a schematic diagram illustrating the process of asynchronously copying candidate blocks to consecutive allocator page groups within the same topology domain according to sorting priority, as provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figures 1 to 3This invention provides a method for monitoring call chain anomalies in a multi-tenant large-model architecture. The technical solution is as follows: A method for monitoring call chain anomalies in a multi-tenant large-scale model architecture, the specific process of which is as follows: Figure 1 As shown, it includes: The main scheduling gateway extracts the tracing context of the inference request and passes it through to the target worker node; The hardware micro-events of the target working node are collected to obtain the actual observation time, physical floating-point operation volume and video memory traffic. The radix tree is parsed to obtain the topological level coordinates of the hit cache block, and then substituted into the preset topology-aware spatial locality decay matrix to calculate the decay penalty factor. The theoretical peak memory bandwidth is calculated by using a decay penalty factor to obtain the topological dynamic memory bandwidth boundary; the physical floating-point operation value, memory flow value and the topological dynamic memory bandwidth boundary are input into the hardware performance boundary model to output the theoretical expected latency baseline. The difference between the actual observation time and the theoretical expected delay baseline is calculated to generate a time-domain delay residual sequence; a discrete wavelet transform is performed on the time-domain delay residual sequence to extract the mean energy of the high-pass subband, which is used as a physical fragmentation index. When the physical fragmentation index exceeds the preset fragmentation threshold, it is determined that a low-level memory access anomaly has occurred in the call chain, an alarm log is generated, and memory consolidation is triggered. Hit cache blocks with non-contiguous logical addresses are extracted as candidate blocks. The sorting priority is calculated based on the communication cost corresponding to the topology level coordinates of the candidate blocks. The candidate blocks are asynchronously copied to the contiguous allocator page group in the same topology domain according to the sorting priority.

[0021] Example 1: Furthermore, the process of passing the tracking context to the target worker node includes: intercepting the inference request containing text data input by the current tenant at the main scheduling gateway; extracting the tenant identifier and a preset service level agreement (SLA) threshold from the inference request, wherein the SLA threshold includes a first-word response time threshold and a single-character generation time threshold; combining the tenant identifier and the SLA threshold to generate the tracking context; establishing a cross-process shared memory channel between the main scheduling gateway and the target worker node; and using the cross-process shared memory channel to map and pass the tracking context to the isolated operating environment within the target worker node.

[0022] When the main scheduling gateway intercepts the inference request containing text data input by the current tenant, the main scheduling gateway continuously listens to the preset external application programming interface port to receive network request packets from the business front end. When the request packet arrives, the protocol parsing module of the main scheduling gateway reads the packet header and payload data, extracts the tenant identity token by parsing the identity authentication field in the request header, and decodes and verifies the token to obtain the unique tenant identifier corresponding to the current request. The decoding and verification specifically adopts a stateless verification mechanism based on JSON Web Token. The main scheduling gateway preloads the public key certificate of the tenant center in its local memory and directly uses the public key certificate to perform local digital signature verification and payload decryption on the extracted tenant identity token, extracting the unique tenant identifier stored in plaintext in the payload, avoiding cross-network database queries during the verification process. Simultaneously, the scheduling configuration field in the packet payload is parsed to extract the service level protocol indicator threshold preset by the business end. This indicator threshold is specifically parsed into two independent time constraint parameters: the first character response time threshold used to limit the pre-filling calculation time, and the single character generation time threshold used to limit the single iteration time of autoregressive decoding.

[0023] After extracting the relevant fields, the tenant identifier and service level agreement (SLA) threshold are combined to generate a tracing context. Specifically, a globally unique request serial number is assigned to the currently intercepted inference request. This request serial number, along with the parsed tenant identifier, first-word response time threshold, and single-character generation time threshold, is assembled into a preset structured data object. Subsequently, the protocol serialization component is invoked to encode the structured data object into a compact binary data stream. The protocol serialization component uses a binary serialization framework based on an interface description language (such as Protocol Buffers or FlatBuffers). Through a predefined static data structure template, the fields in the structured data object are arranged continuously in memory, and redundant metadata such as field names is stripped away. The data is then encoded into a compact binary data stream, thereby generating a tracing context that conforms to the cross-node transmission standard.

[0024] In the process of establishing a cross-process shared memory channel, the main scheduling gateway initiates a system call to the operating system kernel to request the creation of a named shared memory region independent of the standard network protocol stack. This region is then configured with a globally unique name identifier. The main scheduling gateway maps this named shared memory region to its own process address space. Simultaneously, the target worker node obtains this name identifier by listening to an agreed-upon configuration distribution channel and uses it to initiate the same system call, mapping the same named shared memory region to its own process address space. This establishes a cross-process bypass data channel between the two nodes, bypassing the network protocol stack. The agreed-upon configuration distribution channel relies on a key-value storage coordination component (such as ZooKeeper or etcd) of a distributed cluster. After creating the named shared memory region, the main scheduling gateway writes the name identifier as a key-value pair to a designated node path of the coordination component. The target worker node registers a listener with the coordination component, and when it detects a change in the key-value pair under the node path, it retrieves and obtains the name identifier via a callback function.

[0025] When using the cross-process shared memory channel to pass through the tracing context, the main scheduling gateway directly writes the previously serialized binary data stream into the specified offset position of the mapped shared memory region, and sends a synchronization signal to the target worker node to notify that the data is ready. Specifically, the process of sending the synchronization signal is as follows: An operating system-based inter-process communication synchronization primitive, such as an unnamed semaphore or an event file descriptor, is pre-created between the main scheduling gateway and the target worker node; after the main scheduling gateway completes the shared memory write, it performs a V operation (releases resources) on the unnamed semaphore or writes an event count value to the event file descriptor. This triggers an asynchronous interrupt at the system's underlying level, thereby waking up the target worker node that is in a blocked waiting state and indicating that its data is ready. After the target worker node listens to the synchronization signal in its own isolated runtime environment (such as a container isolation environment), it reads the binary data stream from the corresponding offset position in the same shared memory region. The target worker node performs deserialization decoding on the read binary data stream to restore the complete tracking context structured data object, and binds it as metadata to the inference request to be processed. Finally, it pushes it into the request scheduling queue of the large model inference engine in the isolated runtime environment, thereby completing the lossless pass-through of context information.

[0026] By extracting tenant identifiers and service metric thresholds at the main scheduling gateway to generate a tracing context, and then transmitting it to the target node via a cross-process shared memory channel, this design achieves a physical-level binding between tenant identity, service standards, and specific inference requests without modifying the core business logic of the large model engine. This provides a complete data prerequisite for subsequent low-level link monitoring by tenant.

[0027] Furthermore, the process of obtaining the topological level coordinates of the hit cache block from the parsed radix tree includes: starting a non-intrusive probe in the isolated operating environment of the target worker node; dynamically loading the corresponding underlying hardware performance analysis library according to the current hardware chip type of the target worker node; collecting hardware micro-events by listening to the underlying hardware registers through the underlying hardware performance analysis library, and extracting the actual observation time, physical floating-point operation volume, and video memory traffic; and during the initialization phase of the distributed inference system, constructing a static topological level from the logical video memory block to the heterogeneous interconnect architecture using the combination key of the device number and the logical video memory block identifier as an index. The mapping table includes on-chip high-speed interconnect domains, cross-chip switching domains, and cross-host interconnect domains. It periodically collects measured communication delay samples between adjacent working nodes. When the deviation of the measured communication delay sample from the calibration value exceeds a preset drift threshold, it corrects the bandwidth reference value in the static mapping table, generating a dynamic calibration mapping table. When an inference request enters the decoding stage, it parses the cache management status of the large model inference engine in the radix tree, extracts the allocation and release instructions for dynamic key-value cache blocks, obtains the combination key corresponding to the currently hit cache block, queries the dynamic calibration mapping table using the combination key, and extracts the corresponding topology-level coordinates.

[0028] When a non-intrusive probe is launched in the isolated operating environment of the target working node, the probe program is deployed as a background daemon. This probe program first reads the target node's system environment configuration file to identify the underlying hardware chip model of the current node. Based on the identified model, the probe program dynamically searches for and loads a hardware performance analysis shared library that perfectly matches the chip model from a pre-built driver dependency library. After loading, the probe program calls the application programming interface of the underlying hardware performance analysis library to register an event listener callback function with the system's underlying layer to subscribe to the target counter and periodically read the accumulated output from the corresponding underlying hardware register. The probe program collects hardware micro-event streams. Specifically, the underlying hardware registers include a performance monitoring counter for the graphics processor. The probe program reads micro-architecture-level hardware metrics from the performance monitoring counters, such as the total number of active cycles of the streaming multiprocessor, the number of floating-point multiply-accumulate instruction executions, and the L2 and L3 cache read / write byte hit rates, by calling the application programming interface. These metrics form the hardware micro-event stream. The probe program then parses and cleans the collected hardware micro-event streams, extracting the cumulative actual observation time, the total number of physical floating-point operations, and the total amount of video memory read / write data corresponding to the current inference request during execution, as the video memory traffic value.

[0029] During the initialization phase of the distributed inference system, the master node orchestrates and reads the cluster's hardware topology configuration file. For each logical memory block divided in the system, it uses string concatenation to merge the device number of the block with the logical memory block identifier, generating a globally unique key. Subsequently, based on the physical connection hierarchy between nodes recorded in the configuration file, the communication links corresponding to each key are classified and assigned: logical blocks belonging to the same physical chip are assigned to the on-chip high-speed interconnect domain; logical blocks belonging to different chip nodes on the same motherboard are assigned to the cross-chip switching domain; and logical blocks on different working nodes that rely on external network routing devices for communication are assigned to the cross-host interconnect domain. The system uses the key as a retrieval index and persists the corresponding interconnect domain category and rated bandwidth as values ​​to a static mapping table data structure. The initial value of the rated bandwidth is set according to the hardware factory specifications; for example, the on-chip high-speed interconnect domain corresponds to 600GB / s for the NVLink protocol, and the cross-chip switching domain corresponds to PCIe... The 5.0 protocol is set to 128GB / s, and the corresponding RDMA network for cross-host interconnection domains is set to 200Gbps.

[0030] After entering the operation phase, a background timed inspection task is started. The target working node periodically sends probe data packets to other adjacent working nodes in the network topology through a predetermined internal communication protocol, and records the round-trip time from sending the probe data packet to receiving the acknowledgment response. This is used as a sample of measured communication delay. The initial calibrated delay reference value of the corresponding topology level is extracted from the static mapping table, and the absolute value of the difference between the measured communication delay sample and the delay reference value is calculated. When the comparison determines that the absolute value of the difference is greater than the preset system drift threshold, the system triggers the online configuration update process. Based on the current measured communication delay sample, the current actual available communication bandwidth is inferred. The inference process is as follows: extract the preset total byte length of the probe data packet, subtract the inherent photoelectric conversion and link handshake delay constant of the corresponding physical topology layer from the measured communication delay sample to obtain the effective transmission time, then divide the preset total byte length by the effective transmission time to calculate the actual data throughput per unit time, and use the actual data throughput as the current actual available communication bandwidth. The calculated latest available bandwidth value is used to overwrite the original bandwidth reference value in the static mapping table, thereby converting the static mapping table into a real-time dynamic calibration mapping table. Under the premise of the large model inference engine's original binary file, it is synchronized to the memory database of each node for high-frequency querying.

[0031] When an inference request enters the decoding stage of continuous batch processing, the cache scheduling event inside the large model inference engine is intercepted to access the radix tree data structure responsible for managing the dynamic key-value cache. Specifically, the interception and access process employs extended Berkeley Packet Filter (eBPF) or dynamic link library hijacking hook technology. Without modifying the original binary file of the large model inference engine, a probe hook function is injected into kernel mode or user mode to read the memory address pointer. Taking the vLLM inference engine architecture as an example, this includes: using the ptrace system call or the uprobe dynamic tracing tool to attach the probe to the exit of the engine's memory pool allocation initialization function (such as init_cache_engine); intercepting the context structure returned by this function, and reading the root pointer pointing to the dynamic key-value cache management table (i.e., the radix tree structure) based on a pre-determined fixed memory offset (such as offset 0x48 bytes); in this invention, the radix tree is an index structure used to record the prefix sharing relationship of inference requests and the metadata of the hit key-value cache blocks; its node data structure includes at least: The probe uses a 32-bit integer field to record the logical block ID, a 16-bit field to mark the physical GPU device ID, an identifier field to indicate the topology domain, a field to indicate the current cache block reference count, a pointer field to the physical memory base address, and a version number field for state synchronization. Based on the above structure definition, the probe periodically copies and traverses the snapshot of the radix tree through direct memory access or cross-process memory read interfaces (such as process_vm_readv), reads the lifecycle management identifier of each node, parses and extracts the allocation and release instruction operation logs for each dynamic key-value cache block, and when a hit cache block matching the current inference request context prefix is ​​found in the radix tree, the metadata information of the hit cache block is read, the device number and the allocated logical memory block identifier are extracted, and these two pieces of information are assembled into a combination key according to the aforementioned rules. Then, using the combination key as the retrieval parameter, a query request for the dynamic calibration mapping table is initiated to the memory database to read and extract the current exact topology level coordinates and corresponding real-time bandwidth attributes of the hit cache block.

[0032] By combining non-invasive probes with dynamic calibration mapping tables, the exact topological coordinates of the hit cache block on-chip, across chips, or across host levels can be obtained in real time. This allows the system to analyze the memory status without being limited to virtual logical addresses, but to truly reflect the underlying physical communication distance.

[0033] Further, the process of calculating the decay penalty factor includes: reconstructing the memory allocation state using allocation and release instructions; aligning the reconstructed memory allocation state with topological level coordinates; extracting the distribution data of hit key-value cache blocks and their associated cache blocks with topological level coordinate attributes; using the hit key-value cache blocks and their associated cache blocks with parent-child or prefix sharing relationships in the radix tree as matrix nodes; mapping each matrix node to a preset topology-aware spatial locality decay matrix according to its topological level coordinates to generate initial matrix elements; comparing the parent nodes and child nodes that produce logical branches in the radix tree. Based on the corresponding topology level coordinates, determine whether the logical fork crosses a hardware physical boundary with different communication bandwidth levels. When it is determined that the hardware physical boundary has been crossed, extract the current effective bandwidth ratio at both ends of the hardware physical boundary from the dynamic calibration mapping table. Construct an exponential penalty function with the current effective bandwidth ratio as an independent variable. Use the exponential penalty function to perform weight scaling calculation on the initial matrix elements corresponding to the logical fork in the topology-aware spatial locality attenuation matrix, output a nonlinear bandwidth attenuation penalty factor, and use the nonlinear bandwidth attenuation penalty factor as the attenuation penalty factor. The specific process is as follows: Figure 2 As shown.

[0034] In the process of reconstructing the video memory allocation state using allocation and release instructions, a global video memory block status bitmap is first initialized. Following the timestamp sequence, the system sequentially replays the previously extracted allocation and release instructions for dynamic key-value cache blocks. On the status bitmap, the corresponding logical video memory blocks are marked one by one as occupied or free, thus restoring a snapshot of the current video memory allocation. Subsequently, the system extracts the currently hit key-value cache blocks and their associated active block identifiers with parent-child or prefix sharing relationships from the radix tree. These identifiers are then joined with the topology-level coordinates in the dynamic calibration mapping table to generate and extract a distribution data set of hit key-value cache blocks and their associated cache blocks, containing topology-level coordinate attributes such as device number and physical interconnect domain type.

[0035] When generating initial matrix elements, a two-dimensional topology-aware spatial locality decay matrix is ​​instantiated in memory. The row and column indices of this matrix map to the physical memory blocks in the system, respectively. The distribution data set of the physical memory free blocks is traversed. For any two logically related block nodes, a normalized initial spatial proximity value is generated based on their basic weights in the ideal continuous state. This normalized value is filled into the row and column intersection coordinates of the two-dimensional matrix as the initial matrix element of the related block pair. Specifically, the process of generating the normalized initial spatial proximity value includes: extracting the absolute logical address difference between the two logically related block nodes in the global virtual memory allocation table; calculating the ratio of the absolute logical address difference to a preset ideal continuous memory access proximity threshold (such as the standard memory page size of the underlying hardware); and then substituting this ratio into a preset negative exponential decay function to rigorously map the calculation result to a continuous interval between 0 (representing physical limit discreteness) and 1 (representing ideal continuity), thereby generating the normalized initial spatial proximity.

[0036] The specific expression is as follows The independent variable It is the ratio of the absolute logical address difference to the preset ideal consecutive memory access proximity threshold; The attenuation constant is a positive real number. Forced take =1. Wherein, the positive real-valued attenuation constant... This is used to characterize the sensitivity of performance degradation caused by cross-page discrete memory access under a specific underlying hardware architecture. The specific method for obtaining its value is as follows: before the system is officially deployed, an offline benchmark calibration test is performed. Memory allocation snapshots with different discrete distance gradients are pre-constructed on the target working node, and the actual cross-domain communication bandwidth loss rate under each gradient is measured and recorded. Then, the measured data is used as a verification sample set, and the negative exponential decay function is curve-fitted using the least squares method to obtain the corresponding constant value after convergence. In a specific embodiment of this invention, the positive real number decay constant... The preferred value range is 0.1 to 0.5; furthermore, for target working nodes equipped with mainstream graphics processors and high-speed interconnect buses (such as PCIe 5.0), The specific value is set to 0.25.

[0037] In the process of determining whether a logical fork crosses a hardware physical boundary, the system locates the logical fork point that causes the child node split based on the radix tree structure of the large model inference engine. It then extracts the topology level coordinates corresponding to the physical memory block occupied by the parent node and the physical memory block occupied by the child node. The system extracts and compares the interconnect domain category attributes recorded in the two coordinates. If the interconnect domain categories of the parent node and the child node are inconsistent (for example, the parent node is located in the on-chip high-speed interconnect domain, while the forked child node falls into the cross-host interconnect domain), it is determined that the logical fork crosses a hardware physical boundary with different communication bandwidth levels.

[0038] When a crossover is detected, a joint query is initiated to the dynamic calibration mapping table to extract the current effective bandwidth ratio. First, the reference calibration bandwidth value of the on-chip high-speed interconnect domain with the highest communication rate in the entire heterogeneous cluster is extracted. At the same time, based on the topological hierarchy coordinates of the parent node and child node that caused the logical fork, the current actual cross-domain measured bandwidth value between the two is extracted from the mapping table. The reference calibration bandwidth value is used as the numerator, and the cross-domain measured bandwidth value is used as the denominator for division to calculate the current effective bandwidth ratio, which is then used as an independent variable for subsequent penalty calculations.

[0039] During the weight scaling calculation, the system extracts the initial matrix elements corresponding to the logical forks (i.e., the parent and child block pairs that physically cross each other) from the topology-aware spatial locality decay matrix. The system constructs an exponential penalty function with the current effective bandwidth ratio as an independent variable, and uses this exponential penalty function to perform multiplication operations on the initial matrix elements. The specific weight scaling calculation formula is as follows: ; in, This represents the initial matrix element values ​​extracted from the topologically sensed spatial locality decay matrix. This represents the nonlinear bandwidth attenuation penalty factor in the system's calculated output. This represents the preset topology performance sensitivity constant. This represents the current effective bandwidth ratio. This represents a preset nonlinear convergence constant. The specific determination process for the preset topology performance sensitivity constant and nonlinear convergence constant is as follows: Before the system formally receives external inference requests, an offline benchmark calibration task is executed; the system forcibly allocates the large model key-value cache to different topology levels through a preset test script, constructing offline test samples with different physical crossing boundaries; the actual cross-domain communication bandwidth ratio and the corresponding measured performance degradation true value of each offline test sample are measured to form an observation dataset; a nonlinear least squares optimization algorithm is called, using the observation dataset as fitting input parameters, with the mathematical expression of an exponential penalty function as the target fitting curve, iteratively solving for the unknown parameters in the target fitting curve; when the fitting error converges to a preset threshold, the current parameter solution is extracted and solidified as the topology performance sensitivity constant and the nonlinear convergence constant, respectively. The specific form of the exponential penalty function is... In the specific calculation process, if the measured current effective bandwidth ratio If the value is less than 1, it is forcibly reset to 1 to ensure that the current effective bandwidth ratio is always not less than 1; at the same time, the preset nonlinear convergence constant... It is limited to positive real numbers greater than zero to ensure the continuity of mathematical calculations and the clarity of physical meaning.

[0040] The system will calculate the numerical value obtained through the above formula. The output is then written into the local runtime environment of the target worker node as a nonlinear bandwidth attenuation penalty factor that ultimately quantifies the physical communication cost of the cross-domain fork path. This factor can be directly called by subsequent steps to calculate the dynamic memory bandwidth boundary.

[0041] By determining whether a logical fork crosses a physical hardware boundary, the dynamic effective bandwidth ratio is extracted as an independent variable, and an exponential penalty function is constructed for weight scaling. This transforms the abstract cost of physical cross-domain communication into a concrete and calculable attenuation penalty factor value, providing a clear mathematical basis for subsequent calculations of memory bandwidth.

[0042] Further, the process of outputting the theoretical expected latency baseline includes: extracting the factory-preset theoretical peak memory bandwidth of the underlying hardware chip in the target working node, and multiplying the theoretical peak memory bandwidth by the attenuation penalty factor to calculate the topology dynamic memory bandwidth boundary; extracting the theoretical peak computing power value of the underlying hardware chip in the target working node; inputting the physical floating-point operation value, memory traffic value, empirical effective computing power peak value, and topology dynamic memory bandwidth boundary into a preset hardware performance boundary model; in the hardware performance boundary model, calculating a first ratio of the physical floating-point operation value to the empirical effective computing power peak value, and using the first ratio as the computing power-constrained theoretical latency; calculating a second ratio of the memory traffic value to the topology dynamic memory bandwidth boundary, and using the second ratio as the memory access-constrained theoretical latency; extracting the maximum value between the computing power-constrained theoretical latency and the memory access-constrained theoretical latency, generating a theoretical expected latency baseline with fragmentation-constrained characteristics, and using the theoretical expected latency baseline with fragmentation-constrained characteristics as the theoretical expected latency baseline.

[0043] In the process of outputting the theoretical expected latency baseline, the system first reads the system environment configuration file or hardware device firmware information of the target working node to extract the factory-preset theoretical peak memory bandwidth value of the underlying hardware chip within that node. Simultaneously, the system parses the configuration context of the current large model inference engine to identify the quantization precision data type (e.g., half-precision floating-point or 8-bit integer) used in the current inference task. Then, based on the identified quantization precision data type, the system selectively extracts the theoretical peak computing power value matching that data type from the firmware information. When the current inference task is identified as using a mixed precision architecture configuration (e.g., a mixture of INT8 weights and FP16 activation values), the system extracts the basic peak computing power corresponding to the various data types involved in the configuration from the firmware information. Subsequently, based on the large model static computation graph, the system parses the distribution ratio of floating-point operations corresponding to each data type. The extracted basic peak computing power is then weighted and averaged using this distribution ratio, and the calculated comprehensive computing power value is used as the final matched theoretical peak computing power value. Finally, the system extracts the operator effective utilization rate conversion factor, which has been pre-calibrated through offline benchmark testing. The process of obtaining the operator effective utilization conversion coefficient is as follows: Before system deployment, offline performance stress testing is performed on the core operators that are frequently called in the large model inference under an ideal physical continuity state without memory fragmentation; the actual measured computing power throughput of each core operator at the corresponding quantization precision is recorded; the actual measured computing power throughput is divided by the theoretical peak computing power value preset by the underlying hardware chip to obtain the corresponding operator computing power utilization rate; according to the computational load distribution weight of each core operator in the current large model static calculation graph, a weighted average calculation is performed on the computing power utilization rate of each operator to generate and solidify the operator effective utilization conversion coefficient. The system multiplies the theoretical peak computing power value with the operator effective utilization conversion coefficient to calculate the empirical effective computing power peak that can be achieved in engineering. After obtaining the above basic hardware parameters, the system extracts the attenuation penalty factor calculated in the previous step for quantifying the cost of cross-domain communication, performs a multiplication operation between the theoretical peak memory bandwidth and the attenuation penalty factor, and calculates the topological dynamic memory bandwidth boundary after physical fragmentation conversion.

[0044] Subsequently, the system instantiates a preset hardware performance boundary model in the runtime environment. This preset hardware performance boundary model is built upon the underlying roofline theory and is a static boundary evaluation model used to quantitatively analyze the performance bottlenecks of computing tasks under a specific heterogeneous hardware architecture. This model divides the physical execution capability of the underlying hardware into a computing boundary limited by peak computing power and a memory access boundary limited by memory bandwidth. By independently evaluating the approximation of the actual business load to these two physical boundaries, the absolute performance bottleneck of the system is quantitatively identified. The physical floating-point operation volume and memory traffic values ​​corresponding to the current inference request, collected and extracted through non-intrusive probes, along with the newly extracted empirical effective computing power peak and the calculated topological dynamic memory bandwidth boundary, are treated as a complete feature data set and uniformly input into the hardware performance boundary model for extreme value boundary analysis.

[0045] In the analysis and computation flow of the hardware performance boundary model, the system first performs a time-consuming assessment of the computing power dimension. The system uses the input physical floating-point operation value as the dividend and the empirical effective computing power peak value as the divisor to perform a division operation, calculates a first ratio, and confirms this first ratio as the theoretical computing power-constrained time required to complete the current inference task under a purely computing power-constrained ideal state.

[0046] Simultaneously, the system performs a time-consuming evaluation of memory access, using the input video memory traffic value as the dividend and the topology dynamic video memory bandwidth boundary as the divisor to perform a division operation, calculates a second ratio, and confirms this second ratio as the theoretical memory access time required to complete data transfer under the current physical cross-domain and fragmentation-constrained state.

[0047] After completing the time calculations for the above two dimensions, the system calls the numerical comparison logic to compare the generated theoretical time consumption limited by computing power with the theoretical time consumption limited by memory access. Based on the bottleneck effect logic, the system extracts the maximum value from the comparison results as the current absolute performance bottleneck time consumption. By calling the runtime application interface of the underlying hardware, the system extracts the inherent kernel startup overhead constant of the underlying hardware chip in the current target working node executing a single operator task. The absolute performance bottleneck time consumption and the kernel startup overhead constant are added together to perform physical latency compensation calculation. The extracted maximum value is encapsulated to generate a theoretical expected latency baseline with fragmentation-limited characteristics, and this value is output as the final theoretical expected latency baseline for direct use in subsequent monitoring and evaluation steps.

[0048] The dynamic memory bandwidth boundary is derived by multiplying the theoretical peak memory bandwidth by the attenuation penalty factor, and the theoretical latency is calculated under both compute-limited and memory-access-limited conditions. This approach outputs a benchmark reference value that reflects the current reality of physical fragmentation, avoiding evaluation bias caused by using fixed factory parameters.

[0049] Further, the process of using physical fragmentation as an indicator includes: calculating the difference between the actual observation time corresponding to a single decoding and the theoretically expected delay baseline; arranging and concatenating multiple differences within a preset time window in chronological order to generate the time-domain delay residual sequence; constructing a wavelet decomposition filter bank containing a high-pass filter using preset wavelet basis functions; inputting the time-domain delay residual sequence into the wavelet decomposition filter bank for multi-scale discrete decomposition, and extracting the high-pass subband coefficient sequence output by the high-pass filter; and extracting the discrete coefficients of each term in the high-pass subband coefficient sequence. The system calculates the square of each discrete coefficient to obtain the corresponding single-point energy value; it then calculates the arithmetic mean of all single-point energy values ​​within the high-pass subband coefficient sequence to generate the high-pass subband energy mean; it simultaneously collects orthogonal hardware interference micro-events including device temperature downclocking flags, bus retransmission rate, and thread context switching rate, and generates an exclusive verification mask based on a preset interference judgment logic; it then uses the exclusive verification mask to perform multi-source interference filtering on the high-pass subband energy mean to obtain a corrected quantization value characterizing the degree of pure physical memory access jitter, and uses the corrected quantization value as the physical fragmentation index.

[0050] In the process of generating physical fragmentation indicators, the system records the actual observation time corresponding to each decoding operation word by word, and subtracts the actual observation time from the theoretical expected delay baseline output by the previous step to obtain the time difference for each decoding operation. The system extracts multiple consecutive decoding cycles within a preset time window, arranges the multiple time differences involved in the process according to the order of the timestamps, and concatenates them to generate a one-dimensional time-domain delay residual sequence.

[0051] To avoid interference from extreme noise caused by non-low-level memory factors (such as operating system-level thread suspension and context switching) in high-frequency signal extraction, the system performs signal cleaning operations after extracting the time-domain delay residual sequence: First, the global mean of the sequence is calculated and mean-reduction is performed to remove static delay bias; then, the system uses a preset three-standard-deviation (3-Sigma) criterion to perform anomaly truncation on the mean-reduction sequence, smoothly replacing outliers exceeding the standard deviation threshold with local neighborhood means, thereby generating a cleaned standardized residual sequence.

[0052] In the operating environment, the system constructs a wavelet decomposition filter bank containing a high-pass filter using preset wavelet basis functions. The preset wavelet basis functions employ a DoBécy wavelet basis with compact support and orthogonality, specifically the db4 wavelet type. Based on these wavelet basis functions, the system initializes the corresponding high-pass and low-pass filter coefficient matrices, building a multi-scale discrete decomposition logic architecture for signal frequency domain splitting.

[0053] The system inputs the generated standardized residual sequence completely into the wavelet decomposition filter bank. Specifically, within the preset time window, to meet the input signal length requirements of the discrete wavelet transform while also considering real-time monitoring, the preset time window is set to a length of 512 to 1024 consecutive decoding steps (corresponding to approximately 10 to 20 seconds of generation time). When the sequence length requested in a single inference request does not meet the requirements of multi-scale discrete decomposition, the system extracts the beginning and end boundary values ​​of the standardized residual sequence and uses a symmetrical extension method to fill the samples at both ends of the sequence. This eliminates high-frequency truncation artifacts generated when performing boundary convolution on finite-length discrete signals, ensuring the complete frequency domain space required for multi-scale discrete decomposition. After boundary extension, the system performs multi-scale discrete decomposition. In the execution flow of discrete decomposition, the system extracts the signal output branch of the first decomposition level (i.e., Level-1 single-layer decomposition), directionally collects the high-frequency discrete detail coefficients output by the high-pass filter at this level, and reassembles all collected discrete detail coefficients according to the original array timing sequence to extract and generate a high-pass subband coefficient sequence.

[0054] The system iterates through each discrete coefficient in the high-pass subband coefficient sequence, performing a self-multiplication operation on each extracted discrete coefficient to calculate its square value, and recording this square value as the single-point energy value corresponding to that coefficient node. After completing the square operation on all discrete coefficients in the sequence, the system performs a summation operation on all the generated single-point energy values, and divides the summation result by the total number of single-point energy values ​​to calculate the arithmetic mean of the high-pass subband energy. To eliminate high-frequency glitches caused by non-memory factors, the system synchronously collects orthogonal hardware interference micro-events aligned with the preset time window through the underlying driver interface; these orthogonal hardware interference micro-events include the temperature throttling flag of the underlying graphics processor, the retransmission packet loss rate of the peripheral component interconnect bus and the remote direct data access network, and the thread context switching rate of the operating system. The system compares the collected values ​​of various orthogonal hardware interference micro-events with preset system health baseline thresholds. When any value exceeds the corresponding baseline threshold, it is determined that the high-frequency jitter in the current time window is subject to multi-source physical interference, and an exclusive verification mask with a value of zero is generated. When all values ​​do not exceed the corresponding threshold, an exclusive verification mask with a value of one is generated. The system performs a multiplication operation between the exclusive verification mask and the average energy of the high-pass subband to complete the multi-source interference filtering, outputs a corrected quantized value representing the degree of pure physical memory access jitter, and assigns the corrected quantized value to the physical fragmentation index for output and storage.

[0055] The sequence is generated by calculating the difference between the actual time consumption and the theoretical baseline word by word, and the average energy of a single point is obtained by extracting the high-pass subband coefficient sequence using wavelet decomposition filter bank. This calculation step isolates the high-frequency fluctuation characteristics caused by memory fragmentation in the time series, forming a clear quantitative numerical index.

[0056] Further, the process of asynchronously copying candidate blocks to contiguous allocator page groups within the same topology domain according to sorting priority includes: monitoring the running phase of the large model inference engine in the target working node; generating the link anomaly alarm log when the physical fragmentation index is greater than a preset threshold, and determining that the large model inference engine is in a distributed inference communication bubble period, causing the underlying memory bus to be in a low-load idle state, triggering memory regularization; traversing the radix tree, filtering out active cache blocks with internal reference counts greater than zero and non-contiguous physical storage addresses, and using the active cache blocks as candidate blocks; extracting the topology-level coordinates of each candidate block, and extracting the cross-domain communication latency calibration value corresponding to the topology-level coordinates from the dynamic calibration mapping table; forward mapping the cross-domain communication latency calibration value to calculate each of the candidate blocks. The merge reward score of the blocks is used as the sorting priority, and a copy execution queue is generated according to the sorting priority from high to low. A background direct memory access transport stream, independent of the main computing stream, is established in the target working node. Before a candidate block enters asynchronous regularization, the system sets a migration version identifier for the candidate block and prohibits new decoding tasks from binding to the old version address. After confirming the completion of the old version read operation through event synchronization associated with the main computing stream, the background direct memory access transport stream is used to asynchronously copy the candidate block with the highest ranking to the reserved contiguous allocator page group within the same topology domain along the copy execution queue. The addressing pointers of the corresponding copied candidate blocks in the radix tree are redirected to the contiguous addressable logical pages in the contiguous allocator page group within the same topology domain. The specific process is as follows: Figure 3 As shown.

[0057] In the process of asynchronously copying candidate blocks to contiguous allocator page groups within the same topology domain according to sorting priority, a status monitoring service is first deployed in the master node to monitor the running phase of the large model inference engine in the target worker node in real time. It continuously reads the physical fragmentation index output by the preceding steps and compares it numerically with a preset fragmentation threshold in the local configuration file. This preset fragmentation threshold is determined by pre-collecting historical default link samples from the large model inference cluster and extracting the lower quartile of the high-pass subband energy mean using wavelet decomposition. When the physical fragmentation index is determined to be greater than the preset fragmentation threshold... When the fragmentation threshold is reached, the system further extracts the single character generation time threshold from the tracking context and compares the actual observation time corresponding to the current single decoding with the single character generation time threshold. Only when the physical fragmentation index exceeds the limit and the actual observation time synchronously exceeds the single character generation time threshold, the system comprehensively determines that the current multi-tenant call link has experienced a low-level memory access anomaly, and dynamically generates a link anomaly alarm log containing the current tenant identifier, current timestamp, actual observation time, and the value of the physical fragmentation index. The log is then reported to the cluster monitoring center or written to the local persistent log system.

[0058] Simultaneously, the system intercepts the task distribution queue of the inference engine and the underlying hardware performance monitor to identify whether the current computing flow is in the communication bubble period of distributed pipeline parallelism. When it is detected that the system has generated the link anomaly alarm log and determines that the large model inference engine is in the aforementioned communication bubble period, thereby causing the underlying video memory bus to be in a low-load idle state, the system, based on the trigger indication of the link anomaly alarm log, formally issues a video memory consolidation instruction to the runtime environment and starts the online video memory consolidation process.

[0059] Upon receiving the trigger command, the system traverses the radix tree structure recording the video memory allocation status from top to bottom, reading the metadata attributes of each radix tree node one by one. The system uses conditional filtering branches to select active cache blocks with an internal reference count attribute greater than zero and a physical storage state marked as non-contiguous. The system then marks these extracted active cache blocks as candidate blocks for performing video memory merging.

[0060] Subsequently, the system extracts the topological layer coordinates bound to each candidate block and uses these coordinates as a joint query index to extract the cross-domain communication latency calibration value corresponding to the cross-domain physical boundary from a pre-set dynamic calibration mapping table. The extracted cross-domain communication latency calibration values ​​are then normalized through a forward mapping, proportionally mapping the absolute latency time span to a score value within a fixed interval (e.g., 0 to 100). The system calculates the merging benefit score for each candidate block; that is, the higher the cross-domain latency calibration value, the higher the calculated merging benefit score. The system directly uses this calculated merging benefit score as the sorting priority for triggering the copy task and sorts all candidate blocks in descending order according to this priority, generating a unidirectional copy execution queue.

[0061] During physical data transfer, the system invokes the operating system's Direct Memory Access Control (DMI) interface to establish a background DMI transfer stream in the target worker node that is completely independent of the main forward computation stream of the large model. Using this background transfer stream, the system sequentially reads the candidate block data at the top of the queue, following the order of the generated copy execution queue. Without consuming the core computing resources of the main control chip, the system asynchronously copies the key-value cache data within the candidate blocks to a contiguous allocator page group within the same topology domain reserved during system initialization. To avoid bus conflicts caused by background data transfer crossing over to subsequent high-throughput memory access phases, the system calculates the remaining estimated time window of the current communication bubble period in real time and, combined with the underlying video memory bus transmission bandwidth, calculates a dynamic data transfer threshold. During the asynchronous copy, the system accumulates the total amount of data in the transferred blocks. When the total data amount reaches the dynamic data transfer threshold, the system immediately suspends the copy execution queue and forcibly terminates the current batch copy task.

[0062] Specifically, during the distributed inference communication bubble period, before the background direct memory access transport stream performs asynchronous copying, the system calculates the dynamic data transfer volume threshold in real time by constructing a specific mathematical performance mapping model. The specific calculation formula is as follows: ; in, This represents the threshold value for the amount of dynamically transferred data; This represents the estimated remaining time window of the current distributed inference communication bubble period; This represents the measured effective transmission bandwidth of the underlying video memory bus; This represents the conflict avoidance safety boundary coefficient.

[0063] Regarding the remaining estimated time window of the current distributed inference communication bubble period Data Acquisition: The system extracts the data synchronization topology dependencies between the current worker node and its neighboring worker nodes by pre-parsed the distributed pipeline parallel scheduling graph. During the operation of the large model inference engine, the master scheduler monitors the data packet arrival event queue of the underlying network communication interface in real time. When the main computing flow enters the state of waiting for network data synchronization, the system pulls the estimated network transmission delay of the currently pending data packets in the event queue, and subtracts the waiting time consumed at the current moment from the estimated network transmission delay to calculate the remaining estimated time window of the current distributed inference communication bubble period. .

[0064] Measured effective transmission bandwidth of the underlying video memory bus Acquisition: During the initialization and periodic inspection phases of the distributed inference cluster, the target worker node sends a test probe data packet of a preset size on the underlying video memory bus via a background process. The system records the absolute time taken for the test probe data packet to complete bidirectional transmission using direct memory access. The system divides the preset size by the absolute time taken to calculate the real-time throughput. The system directly uses the real-time throughput as the measured effective transmission bandwidth of the underlying video memory bus. This data is then stored in the dynamic calibration mapping table for real-time querying, thereby accurately reflecting the physical status of hardware consumption and bus concurrency usage.

[0065] Regarding the backoff safety boundary coefficient for conflict prevention Determination of: the retreat safety boundary coefficient This is used to establish a time-physical isolation band at the end of the communication bubble period to prevent asynchronous copy operations from spanning to subsequent high-throughput memory access and computation phases. The backoff safety boundary coefficient... The numerical range is limited to between 0.70 and 0.85. Before system deployment, full-load inference is performed under ideal, fragment-free conditions using an offline load testing script. The system iteratively fine-tunes this coefficient and observes the kernel execution latency of the main compute stream. When the fluctuation amplitude of the kernel execution latency is lower than a preset perturbation threshold, the current coefficient value is extracted and solidified as the backoff safety boundary coefficient. In this embodiment, the specific value is 0.80.

[0066] During actual execution of the regularization process, the background direct memory access transport stream sequentially accumulates the physical size of candidate cache blocks along the copy execution queue. When the total accumulated physical size is about to exceed the dynamic data transfer threshold, the system immediately suspends the copy execution queue, thereby ensuring that asynchronous data transfer is completely closed-loop within a safe time window and eliminating the physical risk of memory bus contention between the background regularization and the main computing stream.

[0067] After the data of a single candidate block is asynchronously copied to the contiguous allocator page group within the same topology domain, the system immediately triggers a pointer update operation for the radix tree. The system uses an atomic compare-and-swap instruction to obtain write / modify permissions for the radix tree node corresponding to the candidate block, overwrites the original addressing pointers in the radix tree, and directly redirects them to the newly mapped contiguous physical addresses in the contiguous allocator page group within the same topology domain. After the pointer redirection operation is completed, the system synchronously sends a memory cache refresh barrier instruction to the underlying graphics processor, forcibly invalidating the old L2 cache lines corresponding to the original fragmented block in the relevant streaming multiprocessor, to ensure the consistency of the underlying hardware cache during subsequent high-throughput memory access phases. Subsequently, the system releases the write / modify permissions for the radix tree node; to prevent dangling pointer accesses from parallel computing streams, the system adds the original fragmented physical block to a delayed reclamation queue. After all current read tasks holding the old version identifier have completely exited, the system sends a reclamation instruction to the local memory management unit to release the original fragmented physical block, thus completing a safe and lossless online consolidation closed loop for the candidate block.

[0068] By monitoring the computing power saturation and bus idle status during the pre-filling phase, copying is performed asynchronously using the background direct memory access transport stream, and the reorganization order is determined based on the cross-domain communication latency calibration value. This process prioritizes the contiguous reorganization of physical addresses for scattered blocks with high communication costs without hindering the main computing flow.

[0069] Example 2 In this application scenario, the distributed inference cluster simultaneously hosts online large model services for multiple different tenants. When tenant A (e.g., an online intelligent customer service service) initiates a distributed inference request for a long text conversation, the main scheduling gateway first intercepts the inference request. The gateway's protocol parsing module reads the message payload, extracts the tenant identifier belonging to tenant A and the service level protocol indicator thresholds bound to the service (including the first-word response time threshold and the single-character generation time threshold), and assigns a globally unique serial number to the request, thereby assembling a tracing context. Subsequently, the main scheduling gateway, through a pre-established cross-process shared memory channel, along with a synchronization signal, seamlessly transmits the tracing context to the isolated runtime environment of the target worker node carrying the computation, thus realizing a static benchmark binding between the call chain performance and the tenant identity.

[0070] When tenant A's inference request enters the large model inference engine to perform autoregressive decoding iterative calculation, the non-intrusive probe program deployed in the isolated running environment of the target worker node starts working at high frequency. The probe program dynamically loads the corresponding underlying hardware performance analysis library according to the GPU hardware type of the current node, collects hardware micro-event streams by listening to the performance monitoring register of the graphics processor, cleans and extracts the actual observation time of multi-tenant link calls, physical floating-point operation volume and video memory traffic in the current decoding cycle.

[0071] Simultaneously, the system uses probe hook functions in kernel or user mode to intercept cache scheduling events within the large model inference engine, parsing allocation and release instructions in the KV Cache radix tree management state. By extracting the physical device number and logical block identifier combination key of the currently hit cache block, the system retrieves the calibration mapping table built and executed during system initialization for dynamic online calibration, thereby accurately locating the precise topological level coordinates of the currently hit cache block in the distributed heterogeneous hardware (for example, due to global memory resource shortages, some dynamic key-value cache blocks of tenant A have been forced to be allocated to a cross-host interconnect domain connected by network routing devices).

[0072] Next, the system reconstructs a global memory allocation snapshot using the replayed allocation and release instruction logs. It maps the distribution data of physical memory free blocks with topological hierarchy coordinates to a preset topology-aware spatial locality attenuation matrix. Comparison reveals that the parent and child nodes that logically branch in the current cardinality tree physically cross hardware physical boundaries with different communication bandwidth levels (i.e., from the local on-chip high-speed interconnect domain to the cross-host interconnect domain). The system then extracts the current effective bandwidth ratio at both ends from the dynamic calibration mapping table, substitutes it into an exponential penalty function with this ratio as an independent variable, performs weight scaling calculations, and outputs a nonlinear bandwidth attenuation penalty factor to quantitatively characterize the cost of cross-domain communication.

[0073] After obtaining the penalty factor, the system multiplies it by the theoretical peak memory bandwidth of the underlying hardware chip to calculate the topological dynamic memory bandwidth boundary that conforms to the current physical spatial dispersion state. Subsequently, the hardware performance boundary model (roofline model) receives the input physical floating-point operation volume, memory traffic, empirical effective computing power peak, and the just calculated topological dynamic memory bandwidth boundary. It calculates the theoretical latency due to computing power limitation (first ratio) and the theoretical latency due to memory access limitation (second ratio), respectively. It calls the numerical comparison logic to extract the maximum value of the two, and performs latency compensation in combination with the kernel startup overhead constant. Finally, it outputs the theoretical expected latency baseline of the call chain that conforms to the current physical topology degradation reality.

[0074] Subsequently, the system calculates the difference between the actual observation time and the theoretically expected delay baseline within a single decoding cycle, and then arranges and splices them in chronological order within a preset sliding time window to generate a one-dimensional time-domain delay residual sequence. After being cleaned by demeaning and truncating by three standard deviations, this sequence is completely input into a wavelet decomposition filter bank constructed based on orthogonal Dobermann wavelet basis functions. The system performs Level-1 multi-scale discrete decomposition, directionally acquiring the high-pass subband coefficient sequence output by the high-pass filter to filter out low-frequency trend noise caused by operating system thread suspension, etc. Furthermore, the system sequentially calculates the square of each discrete coefficient in the sequence to obtain the single-point energy value, and calculates the arithmetic mean of all single-point energy values ​​in the entire sequence to generate the high-pass subband energy mean. Subsequently, the system performs exclusive verification filtering by collecting orthogonal hardware interference micro-events such as temperature downclocking, network congestion, and CPU scheduling to exclude artifact jitter caused by sudden changes in the external environment, calculates the exclusive correction quantization value, and uses it as a physical fragmentation index to characterize the risk of abnormal latency caused by pure physical memory fragmentation in the call link.

[0075] At this point, the system's status monitoring service continuously compares this metric. When the output physical fragmentation metric is determined to be greater than the preset fragmentation threshold in the local configuration file, the system further extracts the single-character generation time threshold from the tracking context and compares it with the actual observed time of tenant A. Only when both exceed the limit is the system certain that an implicit memory access anomaly has occurred in the current multi-tenant call chain at the underlying level (i.e., the high fragmentation of the physical memory space has seriously slowed down the upper-layer memory access efficiency and actually caused a service level agreement breach). The system immediately and dynamically generates a link anomaly alarm log containing the tenant A identifier, the current timestamp, and the metric value, and reports it to the cluster monitoring center.

[0076] To eliminate the link anomaly without interfering with the current computation flow, the system continuously intercepts the task distribution queue and hardware status of the large model inference engine. When the system detects that it has generated the link anomaly alarm log and determines that the large model inference engine is in a distributed communication bubble period for subsequent requests (at which time the GPU computing core is waiting for network data synchronization and the underlying PCIe / NVLink memory bus is in a physically low-load idle state), the system, based on the trigger indication of the link anomaly alarm log, formally issues a memory consolidation command to the runtime environment and initiates online link self-healing.

[0077] In the self-healing process, the system traverses the radix tree again, selecting active cache blocks with internal reference counts greater than zero and non-contiguous physical storage addresses as candidate blocks causing the anomaly. The system extracts the topological level coordinates of each candidate block, retrieves the corresponding cross-domain communication latency calibration value from the dynamic calibration mapping table, calculates the merging benefit score of each block through forward mapping, and uses it as the sorting priority, thereby generating a unidirectional copy execution queue from high to low.

[0078] Finally, the system calls the operating system's Direct Memory Access Control (DMI) interface to establish a background DMI transport stream in the target worker node that is completely independent of the forward computation stream of the large model. Using this background transport stream, the system asynchronously moves and copies the key-value cache data in the leading candidate blocks to a reserved contiguous allocator page group within the same topology domain, following the copy execution queue order without intruding on the core computing resources of the main control chip. The total amount moved does not exceed the threshold of the amount of data moved dynamically calculated based on the current pre-filled remaining time window. After the data copy is complete, the system uses atomic compare-and-swap instructions to obtain write / modify permissions for the radix tree node, redirects the addressing pointer to a contiguous addressable logical page in the contiguous allocator page group within the same topology domain, and sends a reclaim instruction for the original fragmented block to the memory management unit. After the reorganization is completed, the actual observed time consumption of tenant A's subsequent decoding cycles significantly decreases and tends to stabilize, the physical fragmentation index drops below the threshold, and the implicit memory access latency anomaly of the multi-tenant large model call chain is successfully monitored and eliminated online without any service interruption.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for monitoring call chain anomalies in a multi-tenant large-scale model architecture, characterized in that, include: The main scheduling gateway extracts the tracing context of the inference request and passes it through to the target worker node; The hardware micro-events of the target working node are collected to obtain the actual observation time, physical floating-point operation volume and video memory traffic. The radix tree is parsed to obtain the topological level coordinates of the hit cache block, and then substituted into the preset topology-aware spatial locality decay matrix to calculate the decay penalty factor. The theoretical peak memory bandwidth is calculated by using a decay penalty factor to obtain the topological dynamic memory bandwidth boundary; the physical floating-point operation value, memory flow value and the topological dynamic memory bandwidth boundary are input into the hardware performance boundary model to output the theoretical expected latency baseline. The difference between the actual observation time and the theoretical expected delay baseline is calculated to generate a time-domain delay residual sequence; a discrete wavelet transform is performed on the time-domain delay residual sequence to extract the mean energy of the high-pass subband, which is used as a physical fragmentation index. When the physical fragmentation index exceeds the preset fragmentation threshold, it is determined that a low-level memory access anomaly has occurred in the call chain, thereby generating an alarm log and triggering memory consolidation. Hit cache blocks with non-contiguous logical addresses are extracted as candidate blocks. The sorting priority is calculated based on the communication cost corresponding to the topology level coordinates of the candidate blocks, and the candidate blocks are asynchronously copied to the contiguous allocator page group in the same topology domain according to the sorting priority.

2. The method for monitoring call chain anomalies in a multi-tenant large-scale model architecture according to claim 1, characterized in that, The process of extracting the tracing context of the inference request through the main scheduling gateway and transmitting it to the target worker node includes: intercepting the inference request containing text data input by the current tenant at the main scheduling gateway; extracting the tenant identifier and a preset service level agreement (SLA) threshold from the inference request, wherein the SLA threshold includes a first-character response time threshold and a single-character generation time threshold; combining the tenant identifier and the SLA threshold to generate the tracing context; establishing a cross-process shared memory channel between the main scheduling gateway and the target worker node; and using the cross-process shared memory channel to map and transmit the tracing context to the isolated operating environment within the target worker node.

3. The method for monitoring call chain anomalies in a multi-tenant large-scale model architecture according to claim 1, characterized in that, The process of obtaining the topological level coordinates of the hit cache block using the parsed radix tree includes: starting a non-intrusive probe in the isolated operating environment of the target worker node; dynamically loading the corresponding underlying hardware performance analysis library according to the current hardware chip type of the target worker node; collecting hardware micro-events by listening to the underlying hardware registers through the underlying hardware performance analysis library, and extracting the actual observation time, physical floating-point operation volume, and video memory traffic; and during the initialization phase of the distributed inference system, constructing a static mapping table from the logical video memory block to the heterogeneous interconnect architecture topological level using the combination key of the device number and the logical video memory block identifier as an index. It includes on-chip high-speed interconnect domain, cross-chip switching domain, and cross-host interconnect domain; it periodically collects measured communication delay samples between adjacent working nodes. When the deviation of the measured communication delay sample from the calibration value exceeds a preset drift threshold, it corrects the bandwidth reference value in the static mapping table and generates a dynamic calibration mapping table; when the inference request enters the decoding stage, it parses the cache management status of the large model inference engine in the radix tree, extracts the allocation instructions and release instructions of the dynamic key-value cache block; it obtains the combination key corresponding to the currently hit cache block, queries the dynamic calibration mapping table with the combination key, and extracts the corresponding topology level coordinates.

4. The method for monitoring call chain anomalies in a multi-tenant large-scale model architecture according to claim 3, characterized in that, The process of calculating the attenuation penalty factor includes: reconstructing the memory allocation state using allocation and release instructions; aligning the reconstructed memory allocation state with topology level coordinates; extracting the distribution data of hit key-value cache blocks and their associated cache blocks with topology level coordinate attributes; using the hit key-value cache blocks and their associated cache blocks with parent-child or prefix sharing relationships in the radix tree as matrix nodes; mapping the topology level coordinates of each matrix node to a preset topology-aware spatial locality attenuation matrix to generate initial matrix elements; comparing the topology level coordinates corresponding to the parent and child nodes of the logical branch in the radix tree to determine whether the logical branch crosses a hardware physical boundary with different communication bandwidth levels; when it is determined that the hardware physical boundary is crossed, extracting the current effective bandwidth ratio at both ends of the hardware physical boundary from the dynamic calibration mapping table; constructing an exponential penalty function with the current effective bandwidth ratio as an independent variable; using the exponential penalty function to perform weight scaling calculation on the initial matrix elements corresponding to the logical branch in the topology-aware spatial locality attenuation matrix to output a nonlinear bandwidth attenuation penalty factor; and using the nonlinear bandwidth attenuation penalty factor as the attenuation penalty factor.

5. The method for monitoring call chain anomalies in a multi-tenant large-scale model architecture according to claim 1, characterized in that, The process of outputting the theoretical expected latency baseline includes: extracting the factory-preset theoretical peak memory bandwidth of the underlying hardware chip in the target working node, and multiplying the theoretical peak memory bandwidth by the attenuation penalty factor to calculate the topology dynamic memory bandwidth boundary; extracting the theoretical peak computing power value of the underlying hardware chip in the target working node, and extracting the operator effective utilization rate conversion factor calibrated in advance through offline benchmark testing, multiplying the theoretical peak computing power value by the operator effective utilization rate conversion factor to calculate the empirical effective computing power peak; and combining the physical floating-point operation value, memory traffic value, empirical effective computing power peak value, and topology... The dynamic memory bandwidth boundary is input into a preset hardware performance boundary model; in the hardware performance boundary model, a first ratio of the physical floating-point operation quantity to the empirical effective computing power peak value is calculated, and the first ratio is used as the computing power-limited theoretical latency; the memory traffic quantity is calculated to a second ratio of the topology dynamic memory bandwidth boundary, and the second ratio is used as the memory access-limited theoretical latency; the maximum value between the computing power-limited theoretical latency and the memory access-limited theoretical latency is extracted to generate a theoretical expected latency baseline with fragmentation-limited characteristics, and the theoretical expected latency baseline with fragmentation-limited characteristics is used as the theoretical expected latency baseline.

6. The method for monitoring call chain anomalies in a multi-tenant large-scale model architecture according to claim 1, characterized in that, The process of using physical fragmentation as an indicator includes: calculating the difference between the actual observation time and the theoretical expected delay baseline corresponding to a single decoding operation word by word; arranging and concatenating multiple differences within a preset time window in chronological order to generate the time-domain delay residual sequence; constructing a wavelet decomposition filter bank containing a high-pass filter using preset wavelet basis functions; inputting the time-domain delay residual sequence into the wavelet decomposition filter bank for multi-scale discrete decomposition and extracting the high-pass subband coefficient sequence output by the high-pass filter; and extracting the discrete coefficients of each item in the high-pass subband coefficient sequence. The square of each discrete coefficient is calculated to obtain the corresponding single-point energy value; the arithmetic mean of all single-point energy values ​​in the high-pass subband coefficient sequence is calculated to generate the high-pass subband energy mean; orthogonal hardware interference micro-events including device temperature downclocking flag, bus retransmission rate and thread context switching rate are collected synchronously, and an exclusive verification mask is generated according to the preset interference judgment logic; the exclusive verification mask is used to perform multi-source interference filtering on the high-pass subband energy mean to obtain a corrected quantization value characterizing the degree of pure physical memory access jitter, and the corrected quantization value is used as the physical fragmentation index.

7. The method for monitoring call chain anomalies in a multi-tenant large-scale model architecture according to claim 3, characterized in that, The process of asynchronously copying candidate blocks to contiguous allocator page groups within the same topology domain according to sorting priority includes: monitoring the running phase of the large model inference engine in the target worker node; generating an alarm log when the physical fragmentation index exceeds a preset threshold, and determining that the large model inference engine is in a distributed inference communication bubble period, causing the underlying memory bus to be in a low-load idle state, triggering memory regularization; traversing the radix tree, filtering out active cache blocks with internal reference counts greater than zero and non-contiguous physical storage addresses, and using these active cache blocks as candidate blocks; extracting the topology-level coordinates of each candidate block, and extracting the corresponding topology-level coordinates from the dynamic calibration mapping table. The cross-domain communication latency calibration value is determined; the cross-domain communication latency calibration value is forward mapped to calculate the merging benefit score of each candidate block, and the merging benefit score is used as the sorting priority. A copy execution queue is generated from high to low according to the sorting priority; a background direct memory access transport stream independent of the main computing stream is established in the target working node; using the background direct memory access transport stream, the candidate blocks ranked first are asynchronously copied to the reserved contiguous allocator page group in the same topology domain along the copy execution queue; the addressing pointers of the corresponding copied candidate blocks in the radix tree are redirected to the contiguous addressable logical pages in the contiguous allocator page group in the same topology domain.

Images