Collaborative simulation method for multiprocessor cluster and computer program product

Abstract

The invention provides a co-simulation method for a multiprocessor cluster and a computer program product. The multi-processor cluster-oriented co-simulation method comprises the following steps of: performing semantic perception modeling on a data stream transmitted across processors so as to mark a semantic attribute for each data object in the data stream; simulating life cycle behaviors of the data object among the plurality of processors based on the semantic attributes, the life cycle behaviors including generation, sharing, failure, and multiplexing; and managing the sequential relationship of each of the lifecycle behaviors through global event scheduling, where the simulation is performed according to the input multiprocessor inference task description, and the performance metrics are output.

Description

Technical Field

[0001] This disclosure relates to co-simulation methods and computer program products for multiprocessor clusters. Background Technology

[0002] With the rapid growth in the size of large language models, the computing power and storage capacity of a single processor are no longer sufficient to meet the demands of inference tasks, making multi-processor cluster deployment the mainstream solution. In distributed inference scenarios, techniques such as tensor parallelism and pipelined parallelism are widely used to split the model's computational tasks across multiple processors for collaborative execution. To evaluate and optimize the performance of multi-processor collaborative inference, simulation tools are widely used in system design, parameter tuning, and other stages. Summary of the Invention

[0003] In one aspect, this application discloses a collaborative simulation method for multiprocessor clusters, comprising: performing semantic-aware modeling on a data stream transmitted across processors to label semantic attributes for each data object in the data stream; simulating the lifecycle behavior of the data object across multiple processors based on the semantic attributes, the lifecycle behavior including generation, sharing, failure, and reuse; and managing the temporal relationships of each behavior in the lifecycle behavior through global event scheduling, wherein the simulation is executed according to an input multiprocessor inference task description and outputs performance metrics.

[0004] In other respects, this application discloses a non-transitory computer-readable medium storing instructions and a computer program product including instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0005] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0006] Figure 1 This is a flowchart of a collaborative simulation method for multiprocessor clusters according to an embodiment of the present disclosure.

[0007] Figure 2 A schematic diagram of the architecture of a multiprocessor cluster collaborative simulation system according to an embodiment of the present disclosure is shown.

[0008] Figure 3 This is a schematic diagram of a computing device that can be implemented according to embodiments of the present disclosure. Detailed Implementation

[0009] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0010] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0011] The phrases “in an embodiment,” “in one embodiment,” and “in some embodiments” are used repeatedly throughout this document. These phrases do not typically refer to the same embodiment; however, they may refer to the same embodiment. Unless the context otherwise specifies, the terms “comprising,” “having,” and “including” are synonyms. The phrases “A or B,” “A and / or B,” and “A / B” mean “(A), (B), or (A and B).”

[0012] With the rapid growth in the size of large language models, the computing power and storage capacity of a single processor are no longer sufficient to meet the demands of inference tasks, making multi-processor cluster deployment the mainstream solution. In distributed inference scenarios, techniques such as tensor parallelism and pipeline parallelism are widely adopted to split the model's computational tasks across multiple processors for collaborative execution. To evaluate and optimize the performance of multi-processor collaborative inference, simulation tools are widely used in system design, parameter tuning, and other stages.

[0013] In such parallel architectures, inter-processor data communication (e.g., the transfer of intermediate activation values, gradients, and cache states) impacts overall performance. However, the inventors recognized that existing simulation frameworks typically abstract inter-processor communication processes into fixed latency or bandwidth models, focusing primarily on rough estimates of communication overhead. In contrast, data flows in real-world multiprocessor systems possess rich semantic information, and different types of data (e.g., cache states, gradient information) exhibit significant differences in lifetime, sharing methods, and consistency requirements. These factors directly affect the system's cache hit rate, communication overhead, and the efficiency of overlapping computation and communication.

[0014] The inventors realized that due to the lack of detailed modeling of data semantics and lifecycle, existing simulation methods are unable to accurately reflect behaviors such as data reuse, cache consistency maintenance, and communication congestion in real systems, resulting in a large deviation between simulation results and actual performance, and failing to provide reliable guidance for system design.

[0015] This disclosure proposes a cross-processor co-simulation framework for multiprocessor clusters. This framework treats cross-processor data flows as a primary simulation object, granting them the same simulation status as computational tasks. By accurately simulating the entire process of key-value (KV) cache, gradients, and other data transmission, compression and decompression, routing, and cache consistency maintenance between processors, this framework can highly reproduce the dynamic execution process of large-scale multiprocessor inference models. This enables predictable evaluation of real-world system performance and provides a reliable basis for the design and optimization of large-scale multiprocessor inference systems.

[0016] The multiprocessor clusters in this disclosure are not limited to a single type of processor, but encompass a variety of computing units suitable for large-scale model inference. For example, the processors may include graphics processing units (GPUs), which, due to their highly parallel architecture and high-bandwidth memory, have become the mainstream hardware for large-scale model inference. For example, the processors may include central processing units (CPUs), suitable for handling control-intensive tasks or lightweight inference scenarios. For example, the processors may also include dedicated artificial intelligence (AI) acceleration chips such as tensor processors and neural network processors, which are deeply optimized for matrix operations and deep learning workloads. For example, the processors may also include reconfigurable or customized computing devices such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). All of the above-mentioned processors or other processors can serve as simulation objects for the co-simulation method described in this disclosure, and this disclosure makes no limitations in this regard. Furthermore, the multiprocessor clusters in this disclosure can be of the same type of processors or heterogeneous processors, and this disclosure makes no limitations in this regard.

[0017] Figure 1 This is a flowchart of a co-simulation method 100 for multiprocessor clusters according to an embodiment of the present disclosure. The method 100 can be executed by a co-simulation system or device for multiprocessor clusters. The co-simulation system or device may include one or more computing devices, such as servers, workstations, personal computers, embedded devices, etc.

[0018] like Figure 1 As shown, method 100 may include operations 110-130. However, in some embodiments, method 100 may include more, fewer, or different operations. This disclosure is not limiting in this respect.

[0019] At point 110, semantic-aware modeling is performed on the data stream transmitted across processors to annotate semantic attributes for each data object in the data stream.

[0020] In some embodiments, semantic attributes may include data types. Data types are used to distinguish data streams of different natures, such as key-value (KV) caches, gradients, and activation values. KV caches are reusable intermediate states during inference; they have a long lifespan and need to be shared across processors, thus requiring tracking of their generation, reuse, and invalidation. Gradients typically occur in training or fine-tuning scenarios, requiring high transmission frequency and consistency. Activation values ​​are intermediate results in forward computation and need to be passed to the next stage processor in pipelined parallelism. In some embodiments, the data types in semantic attributes can be further subdivided into subtypes; for example, KV caches can be distinguished as key caches and value caches, and gradients can be distinguished as model gradients and optimizer state gradients.

[0021] In some embodiments, semantic attributes may include the layer number. The layer number identifies which layer of the model the data object originates from. In a multiprocessor deployment, computational tasks for different layers may be distributed across different processors. Layer number information can be used to determine the destination of the data object and locate the corresponding cache block in the cache directory. It also helps in analyzing the impact of cross-layer data dependencies on pipeline parallel efficiency.

[0022] In some embodiments, semantic attributes may include token ranges. Token ranges are used to indicate which tokens in the input sequence correspond to the data object, for example, represented as start and end token indices, or as a token bitmap. Key-value caches of different tokens may be reused (e.g., in autoregressive generation tasks), and token range information can be used to determine whether a new request can hit an existing cache, thereby achieving key-value cache reuse and avoiding redundant calculations. Furthermore, token ranges are also used to determine the size of the data object and transmission requirements.

[0023] In some embodiments, semantic attributes may include precision format. Precision format describes the numerical representation of a data object, such as FP32, FP16, INT8, etc. Precision format can also describe whether quantization or sparsity is used. Precision affects the size and transmission overhead of data objects, and is also closely related to compression operations. By recording the precision format, the simulation system can accurately calculate the data size, evaluate storage usage and communication bandwidth consumption at different precisions, and provide a basis for compression decisions.

[0024] In some embodiments, semantic attributes may include order attributes. Order attributes are used to indicate the positional information of data objects in a sequence or pipeline. In sequence generation tasks, order attributes can represent the generation order of tokens to manage causal dependencies; in pipeline parallelism, order attributes can represent the stage number of micro-batches in the pipeline to simulate pipeline bubble and compute-communication overlap behavior.

[0025] In some embodiments, semantic attributes may include a request identifier. The request identifier is used to distinguish data objects from different inference requests, the user session ID to which the request belongs, etc. In multi-request concurrency scenarios, the request identifier can isolate the cache space of different requests, avoid data confusion, and can also be used to identify the context of the same request at different processing stages, supporting request-level cache reuse and resource scheduling.

[0026] In some embodiments, semantic attributes may also include timestamp information, such as the creation time of the data object or the expected expiration time, to assist in lifecycle management.

[0027] In some embodiments, semantic attributes may also include a priority label for the data object, indicating the urgency or quality of service requirement of the data object in transmission or cache management.

[0028] In some embodiments, semantic attributes may also include the size or volume information of the data object, which can be used to assist in subsequent transmission scheduling and cache capacity management.

[0029] In some embodiments, semantic attributes may also include the compression status of the data object, such as whether it is compressed or the type of compression algorithm used, so that compression and decompression operations can be handled correctly during simulation.

[0030] In some embodiments, semantic attributes may also include a source processor identifier and / or a target processor identifier for data objects, used to track the flow path of data across multiple processors.

[0031] In some embodiments, different combinations of semantic attributes can be used to define different categories of data objects. For example, the combination of "data type=KV cache", "level number=12", and "precision format=INT8" can be used to identify a quantized KV cache at level 12.

[0032] In some embodiments, semantic attributes may remain constant throughout the lifecycle of a data object or may be dynamically updated when specific events (such as compression or format conversion) occur.

[0033] In some embodiments, semantic-aware modeling can be implemented by adding metadata tags containing semantic attributes to the header of a data object. In some embodiments, semantic-aware modeling can be implemented by maintaining a separate metadata structure associated with the data object, which is mapped to the data object using a unique identifier. In some embodiments, semantic-aware modeling can be performed on-the-fly when the data object is generated, or when the data object is first transferred across processors. In some embodiments, semantic-aware modeling can be performed by a semantic-aware interface within a processor emulation node, which automatically adds or resolves semantic attributes as the data object leaves or enters the processor.

[0034] At point 120, based on semantic attributes, the lifecycle behavior of data objects across multiple processors is simulated. For example, the simulation system maintains a lifecycle state machine for each data object, which drives state transitions based on the data object's semantic attributes (e.g., data type, layer number, token range, request identifier, etc.) and system events (e.g., computation task completion, cache full, new request arrival).

[0035] In some embodiments, lifecycle behaviors may include generation, sharing, invalidation, and reuse.

[0036] For the generation behavior, when the processor performs a computational task and generates a new data object (such as a key-value cache block or intermediate activation value), the simulation system creates a new record in the global cache directory based on the semantic attributes of the data object. This record contains the data object's unique identifier, the processor it belongs to, the generation timestamp, the storage location, and the initial reference count. Generation behavior typically occurs during the pre-population phase or the forward computation process.

[0037] For shared behavior, when a data object needs to be accessed by multiple processors simultaneously (e.g., in tensor parallelism or pipelined parallelism), the simulation system tracks its sharing relationships by maintaining a reference count for the data object. The reference count increments whenever a processor requests access to the data object and decrements when the processor finishes accessing and releasing the data. Changes in the reference count can be used to determine whether the data object is still in an active shared state. In some embodiments, the triggering condition for shared behavior is based on the data type and layer number in the semantic attributes: for example, key-value caches, due to their reusability, often have longer sharing cycles and higher sharing frequencies.

[0038] For reuse behavior, when a new inference task arrives, the simulation system queries the global cache directory based on the semantic attributes of the new task (e.g., request identifier, token range) to check if there are any reusable data objects. For example, if the request identifier of the new task is the same as the request identifier of a KV cache block in the directory, and the token ranges overlap or are consecutive, it is determined to be reusable. In this case, the simulation system directly marks the data object as being used by the current task and updates its access timestamp and reference count, avoiding repeated generation and transmission processes.

[0039] For failure events, the simulation system triggers failure events based on a preset cache eviction policy. The cache eviction policy can be configured differently based on the semantic attributes of the data objects: for example, a Least Recently Used (LRU) policy can be used for key-value (KV) caches; a First-In-First-Out (FIFO) policy can be used for gradient data. When the cache capacity reaches its limit or the data object is no longer needed, the system selects the data object to be invalidated according to the chosen policy, removes it from the global cache directory, and triggers a corresponding failure notification. In some embodiments, the failure behavior also considers the consistency state of the data object: for example, modified data objects need to undergo a write-back operation before becoming invalid.

[0040] In some embodiments, lifecycle behaviors may further include at least one of the following: migration, write-back, invalidation, prefetching, etc. Migration is used to move a data object from the local storage of one processor to the local storage of another processor, for example, triggered in load balancing or fault recovery scenarios. Write-back is used to write modified data objects from the processor cache back to main memory or shared storage, for example, performed when cache eviction or consistency protocol requirements are met. Invalidation is used to mark a data object as invalid, making it inaccessible or unreusable, for example, occurring during write invalidation operations in a consistency protocol or explicit cache flushing. Prefetching is used to load a data object from a remote processor or main memory into the local cache in advance before it is actually needed, thus hiding access latency.

[0041] In some embodiments, simulating the lifecycle behavior of data objects across multiple processors also includes simulating the execution process of cache coherence protocols to manage the consistency state of shared data objects across multiple processors. As mentioned earlier, the simulation system maintains a reference count for each data object through a global cache directory to track its sharing relationships among processors. For example, when the reference count is greater than 1, it indicates that the data object is currently being shared by multiple processors simultaneously. In this case, a cache coherence protocol needs to be introduced to coordinate the access and modification of the same data object by each processor. Based on this, the simulation system simulates the execution process of different types of cache coherence protocols according to the semantic attributes of the data object and the current access mode. The cache coherence protocols include, for example, write-update protocols, write-invalidate protocols, or directory-based protocols.

[0042] For simulations of write-update protocols, in some embodiments, when a processor modifies a data object shared by multiple processors, the simulation system first generates an update message containing the modified data. Then, the system queries the global cache directory for a list of all processors holding copies of the data object and broadcasts the update message to these processors. The processor simulation nodes that receive the update message update their local cache copies accordingly. Simultaneously, the status information of the data object in the global cache directory is updated to "modified" or remains in the "shared" state (depending on the specific design of the protocol).

[0043] In simulations of write invalidation protocols, in some embodiments, when a processor modifies a shared data object, the simulation system does not send a complete data update. Instead, it sends an invalidation message to all processors holding copies. Upon receiving the invalidation message, these processors mark their local cached copies of the data as invalid. When other processors subsequently need to access the data object, they request the latest data from the processor or main memory that has the latest version, based on the cache miss mechanism. At this time, the simulation system tracks the cache state transitions of each processor after the invalidation operation based on the reference count changes recorded in the global cache directory, simulating the complete lifecycle process from "shared" to "invalid" and then to "re-fetch".

[0044] For the simulation of directory-based protocols, in some embodiments, the simulation system uses a global cache directory as a centralized data structure for consistency maintenance. It maintains a directory entry for each shared data object, recording the cache state of each processor (e.g., "shared," "exclusive," "modified"). When a processor initiates an access to or modification request for a data object, the simulation system first queries the directory entry and determines the next operation based on the current state and request type. For example, when a processor requests to modify a data object in the "shared" state, the system needs to first send an invalidation message to all sharers. After receiving confirmation, it changes the state to "modified" before allowing the modification operation.

[0045] At point 130, the temporal relationships of the behaviors within the lifecycle behavior are managed through global event scheduling. In some embodiments, managing the temporal relationships of the behaviors within the lifecycle behavior through global event scheduling may include: assigning timestamps to the behaviors within the lifecycle behavior and executing them sequentially to simulate synchronous waiting, overlapping computation and communication, and pipelined parallel behavior in multiprocessor cooperative inference.

[0046] In some embodiments, managing the temporal relationships of behaviors within a lifecycle behavior through global event scheduling can employ a discrete event simulation mechanism based on event queues and global timestamps. For example, the simulation system maintains a global event queue, where each event corresponds to a specific action within a lifecycle behavior (e.g., a generation, sharing, reuse, or failure event), and assigns a unique timestamp to each event. The global event scheduler retrieves events from the queue in ascending order of timestamps and executes them, ensuring that all behaviors are accurately simulated in the chronological order of their occurrence. When a new event is triggered (e.g., due to the completion of a computational task or the arrival of a message), the global event scheduler generates a timestamp for it and inserts it into the correct position in the event queue.

[0047] In some embodiments, the global event scheduler also maintains dependencies between events to handle sequence constraints arising from data or control dependencies. For example, the reuse of a data object can only occur after its generation is complete; failure may depend on all processors sharing the data object having released their references. The global event scheduler manages these constraints through a dependency graph or a list of prerequisite events, allowing an event to execute or releasing it from a waiting state only when all its prerequisites are met.

[0048] In some embodiments, to accurately simulate the overlap between computation and communication, the global event scheduler supports the parallel scheduling of computational and communication events. For example, while a processor is performing a computational task, the global event scheduler can simultaneously schedule transmission events of data objects unrelated to that processor on the interconnect network. By assigning different timelines to computational and communication events or allowing them to interleave in time, the simulation system can realistically reflect the parallel operation of computational and communication units in a real system. For example, when processor A performs matrix multiplication, processor B can simultaneously send KV buffers to processor C; the global event scheduler ensures that the timestamps of these two types of events do not conflict and reflects the overlap effect during execution.

[0049] In some embodiments, to simulate bubble-filling behavior in parallel pipelines, the global event scheduler employs a micro-batch scheduling strategy. The simulation system divides input requests into multiple micro-batches and assigns computation events to each micro-batch at various stages of the pipeline. The global event scheduler accurately calculates the start and end times of each micro-batch based on its arrival time at each stage and the computation time of each stage, thereby identifying idle periods (i.e., bubbles) in the pipeline caused by incomplete preceding stages. By adjusting the size or scheduling order of the micro-batches, users can evaluate the impact of different pipeline configurations on overall throughput and latency.

[0050] In some embodiments, the global event scheduler supports the simulation of various synchronization primitives, including barriers, locks, and semaphores. When a synchronization operation is encountered during simulation, the global event scheduler suspends the execution of the relevant processor and places it in a waiting queue until the synchronization condition is met. For example, in tensor parallelism, all processors participating in the computation need to perform barrier synchronization after each computation step; the global event scheduler collects barrier arrival events from all processors, and only releases all waiting processors to continue execution when the last processor arrives. This accurate synchronization simulation helps to evaluate the impact of synchronization overhead on overall performance.

[0051] In some embodiments, the global event scheduler also supports dynamic priority scheduling, for example, assigning different priorities to different types of actions based on the semantic attributes of data objects. For instance, latency-sensitive key-value buffer transmissions can be assigned a higher priority, allowing them to take precedence over lower-priority gradient transmission events in the event queue. While executing events in timestamp order, the global event scheduler allows high-priority events to preempt the execution window of low-priority events, thereby simulating QoS guarantee mechanisms in real-world systems.

[0052] In some embodiments, the global event scheduler works in conjunction with the interconnect network simulation module to handle the impact of network congestion on event timing. When network congestion leads to increased transmission delays, the completion timestamp of a communication event is postponed accordingly, affecting the start time of other actions that depend on that communication event (e.g., subsequent computational tasks). The global event scheduler reflects this coupling effect by dynamically adjusting the event timestamps, ensuring that simulation results accurately reflect the impact of network state changes on overall performance.

[0053] In some embodiments, the global event scheduler also supports time warp or optimistic synchronization mechanisms to accelerate simulation execution. When a causal error is detected, the global event scheduler can roll back to a previous state and re-execute to ensure the correctness of the simulation. This mechanism is suitable for scenarios requiring rapid exploration of large-scale design spaces.

[0054] Through the above embodiments, the global event scheduler can flexibly and accurately manage the temporal relationships of each behavior in the lifecycle behavior, thereby achieving high-fidelity simulation of complex temporal behaviors (e.g., synchronous waiting, computation-communication overlap, pipeline parallelism) in multiprocessor collaborative reasoning.

[0055] The simulation described above executes based on the input multiprocessor inference task description and outputs performance metrics. In some embodiments, performance metrics may include end-to-end latency, throughput, processor utilization, communication percentage, cache hit rate, etc. In some embodiments, performance metrics may also include at least one of the following: average response time, tail latency, synchronization wait time, pipeline bubble time, peak memory usage, cache miss rate, cache reuse distance, number of consistent messages, total number of communication messages, routing hop distribution, parallel efficiency, load imbalance, compression ratio, and communication time saved by compression. These metrics characterize the performance features of multiprocessor collaborative inference from different dimensions, providing a more comprehensive reference for system optimization.

[0056] In some embodiments, the simulation system explicitly models data compression and decompression operations during the simulation of lifecycle behavior to evaluate the performance trade-offs across the entire compression-transmission-decompression link. This modeling process is associated with stages such as sharing and transmission in the lifecycle behavior. For example, when data objects need to be transmitted across processors due to sharing behavior, the simulation system can trigger a compression operation to reduce the amount of data transmitted and trigger a corresponding decompression operation at the receiving end.

[0057] In some embodiments, explicit modeling of data compression and decompression operations includes configuring the compression algorithm type and compression ratio. The simulation system supports parameterized configuration of various compression algorithms, such as quantized compression, sparse compression, low-rank decomposition, or entropy coding. Each compression algorithm type corresponds to a predefined range of compression ratios, and specific compression algorithms and compression ratio parameters can be selected as needed. Subsequently, the simulation system determines the computational overhead of compression and decompression operations based on the configured compression algorithm type and compression ratio. The simulation system can have built-in computational overhead models for different compression algorithms, which are constructed based on algorithm complexity analysis or measured data. The computational overhead is quantified in processor cycles or microseconds and can be dynamically adjusted according to the processor's computing power parameters. The simulation system calculates the compressed data volume based on the compression ratio and the original data size, and then estimates the communication time saved after compression by combining the bandwidth parameters of the interconnection network model.

[0058] Based on this, the simulation system compares the additional computation time introduced by compression and decompression operations with the communication time saved by compression to evaluate the performance trade-offs of the entire compression-transmission-decompression link. For example, the simulation system compares the computation time introduced by compression and decompression with the communication transmission time saved due to the reduction in data size. If the net benefit of compression is positive, it indicates that using compression helps improve end-to-end performance under current network conditions and computing resources; if it is negative, the overhead introduced by compression outweighs the benefit of communication savings. The simulation system can try multiple compression algorithms and compression ratio combinations in a single simulation run, outputting the net benefit of each combination for the user to choose the optimal configuration.

[0059] In some embodiments, compression modeling is also combined with the semantic attributes of data objects to achieve differentiated compression strategies. For example, for KV caches, which are sensitive to accuracy and may be reused multiple times, a lower compression ratio or lossless compression can be used; for gradients, which are sparse during training, a high compression ratio strategy such as Top-K sparsification can be used; for activation values, the compression intensity can be dynamically adjusted according to their layer number, such as using a low compression ratio for sensitive layers and a high compression ratio for non-sensitive layers.

[0060] In some embodiments, compression operations can be performed at different nodes along the transmission path, including compression on the sending processor before transmission, compression on intermediate devices such as switches or smart network interface cards (NICs), or decompression on the target processor before use. The simulation system can model the attribution of computational overhead and changes in the transmission path based on the location of compression execution. Furthermore, the simulation system can support compression-aware routing strategies, dynamically selecting the transmission path based on whether the data object is compressed and its compression ratio. For example, small-volume data with a high compression ratio can take a latency-sensitive but bandwidth-limited fast path, while large-volume uncompressed data can take a high-bandwidth path.

[0061] Through the explicit modeling described above, the simulation system can accurately evaluate the impact of different compression strategies on end-to-end performance, including whether the communication time saved by compression is sufficient to offset the computational overhead, the potential impact of compression on cache reuse efficiency, the performance of different compression algorithms under different network loads, and the impact of the additional latency introduced by compression on bubble-filling behavior in pipeline parallelism. Incorporating compression and decompression operations as explicit computation-communication coupling links into the simulation avoids the shortcomings of traditional simulations that simplify them to fixed overhead or ignore them. This allows users to make better compression strategy choices based on simulation results, finding the optimal balance between computational overhead and communication savings.

[0062] In some embodiments, during the simulation of lifecycle behavior, when data objects undergo sharing behavior and need to be transferred across processors, the simulation system simulates the transfer process of data objects between multiple processors based on a multi-level interconnect network model. This transfer process simulation works in conjunction with the lifecycle management module: once the sharing behavior is triggered (e.g., a processor requests access to a data object on another processor, and the reference count increases), the simulation system invokes the interconnect network model to simulate the complete data transfer path from the source processor to the target processor, including packet segmentation, routing, link transmission, and possible congestion waiting.

[0063] For example, a multi-level interconnect network model is a hierarchical modeling simulation component used to accurately characterize the complex interconnect structures in modern multiprocessor clusters. This model includes parameterized descriptions of various physical links and switching devices, such as NVLink interconnect technology, PCIe (High-Speed ​​Peripheral Interconnect), InfiniBand, Ethernet, and switches. For each interconnect technology, the model can define its link bandwidth (e.g., unidirectional bandwidth of NVLink 3.0, transmission rate of PCIe Gen5), transmission delay (including link propagation delay and switching node processing delay), and link-layer protocol characteristics (e.g., flow control, error retransmission mechanisms).

[0064] In some embodiments, the multi-level interconnected network model may also include an explicit description of the topological connections. The simulation system supports defining various common cluster topologies, such as fully interconnected (all processors are directly connected to each other), tree (a hierarchical structure with switches at its core), ring, fat-tree, or torus. Users can specify the connection methods between processors and switches, the number of links, and the bandwidth and latency parameters of each link through configuration files or a graphical interface. This topology description provides the basis for subsequent routing calculations and congestion simulations.

[0065] In some embodiments, the multi-level interconnection network model supports the simulation of multi-hop routing. When there is no direct physical link between the source processor and the destination processor, data objects need to be forwarded through one or more intermediate switching nodes. The simulation system calculates the complete forwarding path for each packet according to a preset routing policy (e.g., static routing table, destination address-based dynamic routing, or shortest path first algorithm), and simulates route lookup, queuing, and forwarding latency at each node on the path. For networks that support adaptive routing, the model can also dynamically adjust path selection based on the current link load to simulate load balancing behavior in real hardware.

[0066] In some embodiments, the multi-level interconnected network model also includes simulations of network congestion effects. Congestion effects are primarily implemented through a link queue model based on queuing theory. For example, each link or switching node port is associated with a queue of finite depth. When multiple data streams simultaneously compete for the same link, packets wait in the queue, leading to increased transmission latency. The simulation system can dynamically calculate the queuing delay for each packet based on link bandwidth, packet arrival rate, and queue length, and simulate packet loss or backpressure mechanisms when the queue overflows. Congestion effects further impact end-to-end latency in multi-hop routing and may trigger retransmissions or flow control in upper-layer protocols, thus more realistically reflecting performance changes in real-world networks.

[0067] In some embodiments, the interconnection network model can also support assigning different transmission priorities and quality of service (QoS) levels based on the semantic attributes of data objects, i.e., achieving QoS differentiation. For example, latency-sensitive key-value (KV) buffer transmissions can be assigned higher priorities, allowing them to jump the queue in the link queue or occupy more bandwidth; while background gradient synchronization can be assigned lower priorities, thus ensuring high-priority traffic is prioritized during congestion. The simulation system simulates this priority differentiation by configuring different queue scheduling algorithms (e.g., strict priority, weighted fair queue) and evaluates its impact on end-to-end performance.

[0068] Through the aforementioned multi-level interconnected network model, the simulation system can accurately calculate the end-to-end latency of data objects transmitted between multiple processors, including link propagation latency, switching node processing latency, cumulative latency introduced by routing hop count, and queuing latency caused by congestion. This latency data, combined with timestamp management in lifecycle behavior, enables the global event scheduler to accurately schedule computation or synchronization events that depend on transmission completion, thereby achieving realistic simulation of behaviors such as computation-communication overlap, pipelined parallelism, and synchronization blocking.

[0069] Figure 2 A schematic diagram of the architecture of a multiprocessor cluster collaborative simulation system 200 according to an embodiment of the present disclosure is shown. Figure 2 As shown, the system 200 includes multiple processor emulation nodes (e.g., GPU0, GPU1, ..., GPUn), an interconnect and routing emulation module 210, a global cache and consistency manager 220, and a cluster-level time and event scheduler 230. Each processor emulation node simulates a physical processor (e.g., a graphics processing unit), internally containing a streaming multiprocessor (SM) and a Level 2 cache (L2), and is configured with a key-value cache storage area for storing key-value cache data generated during inference. The processor emulation nodes interact with each other through the interconnect and routing emulation module 210, which simulates a multi-level interconnect network, including physical links and switching devices such as NVLink, PCIe, Switch, and Network on Chip (NoC).

[0070] The Global Cache and Consistency Manager 220 is bidirectionally connected to the Interconnect and Routing Simulation Module 210. It maintains a global cache directory across processors, recording information such as the storage location, reference count, and consistency status of each data object on each processor, and manages the execution of cache consistency protocols (e.g., write updates, write invalidations, or directory-based protocols). The Cluster-Level Time and Event Scheduler 230 is connected to all the aforementioned modules (including the processor simulation nodes, the Interconnect and Routing Simulation Module 210, and the Global Cache and Consistency Manager 220). It is responsible for assigning global timestamps to all events during the simulation process (e.g., computation start, data transfer, cache invalidation, synchronization barriers, etc.) and driving event execution in chronological order to accurately simulate the timing relationships in multi-processor collaborative inference.

[0071] In this architecture, when processor simulation nodes generate or access data objects, they annotate each data object with semantic attributes (e.g., data type, layer number, token range, etc.) through a data semantic awareness interface, and pass this information along with the data object to the interconnect and routing simulation module 210 and the global cache and consistency manager 220. The interconnect and routing simulation module 210 selects an appropriate routing path based on the semantic attributes of the data object and the current network state, and simulates link transmission delays and potential congestion effects. The global cache and consistency manager 220 updates the cache directory based on the semantic attributes and lifecycle state of the data object, triggers cache consistency operations, and notifies the cluster-level time and event scheduler 230 of relevant events. The cluster-level time and event scheduler 230 integrates events from all modules, maintains a global event queue, and ensures that all actions proceed in causal order and with timestamps, thereby realistically reproducing behaviors such as synchronous waiting, computation-communication overlap, and pipeline parallelism in a multiprocessor environment.

[0072] In one embodiment, based on Figure 2 The system architecture shown illustrates that the process of performing a multiprocessor collaborative inference simulation may include the user first inputting a description of the multiprocessor inference task into the system 200, including the model structure, parallelism strategy (such as tensor parallelism, number of pipeline stages), and request load characteristics. The cluster-level time and event scheduler 230 initializes the global timestamp and starts the simulation loop.

[0073] Suppose that during the simulation, GPU0 executes a computational task at a certain layer, generating a new KV cache data object. The simulation node of GPU0 annotates the semantic attributes of this data object, including data type = "KV cache", layer number = 5, token range = 10-20, precision format = FP16, etc., and sends the information of this data object to the global cache and consistency manager 220. The manager records the generation time, storage location (GPU0), and initial reference count of the data object in the global cache directory, and reports the "generation event" to the cluster-level time and event scheduler 230.

[0074] Subsequently, GPU1 needs to access the KV cache data object to process subsequent tokens for the same request. At this time, GPU1 initiates a data request to GPU0 through the Interconnect and Routing Simulation Module 210. The Interconnect and Routing Simulation Module 210 calculates the transmission path and latency based on the current network topology (e.g., direct connection via NVLink) and link load, and simulates the data packet transmission process. Simultaneously, the Global Cache and Consistency Manager 220 detects this sharing behavior, adds GPU1 to the sharer list of the data object, and increments the reference count. The manager notifies the cluster-level time and event scheduler 230 of the "sharing event" and related transmission completion events.

[0075] The cluster-level time and event scheduler 230 rationally schedules the computation and data transmission of GPU0 and the subsequent computation of GPU1 based on the timestamps of each event, so that the simulation can accurately reflect the overlap of computation and communication. For example, during data transmission, the cluster-level time and event scheduler 230 allows GPU0 to continue to execute other unrelated computations, thereby simulating the pipeline overlap effect in a real system.

[0076] Once the request is processed, the global cache and consistency manager 220 determines that the KV cached data object can be invalidated according to a preset cache eviction policy (e.g., LRU), triggers an invalidation event, and removes the record from the cache directory. The cluster-level time and event scheduler 230 records the invalidation time and updates the cache status of the relevant processors. Finally, after the simulation runs, the system 200 outputs performance metrics such as end-to-end latency, processor utilization, communication ratio, and cache hit rate.

[0077] This disclosure proposes a collaborative simulation scheme for multiprocessor clusters. By semantically aware modeling the data flow transmitted across processors, the simulation system can identify and track the semantic attributes of data objects. Based on this, it simulates the complete lifecycle of data objects across multiple processors, including behaviors such as generation, sharing, failure, and reuse. Furthermore, it precisely manages the temporal relationships of each behavior through global event scheduling. In addition, the collaborative simulation scheme of this disclosure can also integrate multi-level interconnection network transmission modeling and compression-decompression performance evaluation. These technical solutions organically integrate data semantics, lifecycle, interconnection network, and time scheduling, overcoming the accuracy deviations caused by existing simulation frameworks neglecting data semantics and lifecycle, and achieving high-fidelity simulation of the entire process of multiprocessor collaborative inference. The technical solutions of this disclosure can output performance indicators such as end-to-end latency, throughput, processor utilization, communication ratio, and cache hit rate, providing reliable quantitative basis for the design optimization, resource allocation, and algorithm selection of large-scale inference systems (e.g., distributed systems), significantly improving the accuracy and predictability of performance evaluation.

[0078] Figure 3 This is a schematic diagram of a computing device that can be implemented according to embodiments of this disclosure, and can be used to implement the collaborative simulation system or device for multiprocessor clusters described above. For example... Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more hardware processors 304 coupled to bus 302 for processing information. The one or more hardware processors 304 may include, for example, one or more general-purpose microprocessors.

[0079] like Figure 3As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302. Main memory 306 is used to store information and instructions executed by one or more processors 304, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. These instructions, when stored in storage media accessible to one or more processors 304, can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0080] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0081] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0082] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0083] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0084] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0085] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0086] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0087] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0088] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0089] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0090] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0091] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0092] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0093] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0094] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the term "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A collaborative simulation method for multiprocessor clusters, characterized in that, include: Semantic-aware modeling is performed on data streams transmitted across processors to label semantic attributes for each data object in the data stream; Based on the aforementioned semantic attributes, the lifecycle behavior of the data object across multiple processors is simulated, including generation, sharing, invalidation, and reuse; and The temporal relationships between the behaviors in the lifecycle are managed through global event scheduling. The simulation is executed based on the input multiprocessor inference task description and outputs performance metrics.

2. The method according to claim 1, characterized in that, The semantic attributes include at least one of the following: data type, layer number, token range, precision format, order attribute, request identifier, priority label, size or volume information, compression status, source processor identifier, and target processor identifier.

3. The method according to claim 2, characterized in that, The data types include key-value caches, gradients, or activation values.

4. The method according to claim 1, characterized in that, Also includes: During the simulation of the lifecycle behavior, data compression and decompression operations are explicitly modeled to evaluate the performance trade-offs of the entire compression-transmission-decompression link.

5. The method according to claim 4, characterized in that, Explicit modeling of data compression and decompression operations includes: Configure the compression algorithm type and compression ratio; Based on the compression algorithm type and compression ratio, determine the computational overhead of compression and decompression operations; and The computational overhead is compared with the time saved by the compressed communication to evaluate the performance trade-off of the entire compression-transmission-decompression link.

6. The method according to claim 1, characterized in that, Also includes: When sharing occurs during the lifecycle behavior, the data object is simulated for transmission between multiple processors based on a multi-level interconnection network model.

7. The method according to claim 6, characterized in that, The multi-level interconnection network model is used to simulate the link bandwidth, transmission delay, topology connection relationship, multi-hop routing, and network congestion effects of NVLink, PCIe, InfiniBand, Ethernet, or switches.

8. The method according to claim 1, characterized in that, The simulation of the lifecycle behavior of the data object across multiple processors includes: For the generation behavior, the generation time and storage location of the data object are recorded through a global cache directory; For shared behavior, the sharing relationship between the data objects is tracked by maintaining the reference count of the data objects; For reuse behavior, the system checks whether a new task can reuse an existing data object based on the global cache directory; and For invalid behavior, an invalidation event is triggered based on a preset cache eviction policy.

9. The method according to claim 8, characterized in that, Simulating the lifecycle behavior of the data object across multiple processors also includes: The execution process of a cache coherence protocol is simulated, which includes a write-update protocol, a write-invalidate protocol, or a directory-based protocol.

10. The method according to claim 1, characterized in that, Managing the temporal relationships of various behaviors within the lifecycle through global event scheduling includes: Timestamps are assigned to each behavior in the lifecycle behavior and executed sequentially to simulate synchronous waiting, overlapping computation and communication, and pipelined parallel behavior in multiprocessor collaborative inference.

11. The method according to claim 1, characterized in that, The performance metrics include end-to-end latency, throughput, processor utilization, communication percentage, or cache hit rate.

12. The method according to claim 1, characterized in that, The processor includes a graphics processor.

13. A computer program product, comprising instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-12.

Images