A distributed computing scheduling method and management system for attention model

Abstract

This invention discloses a distributed computing scheduling method and management system for attention models. The method includes: the operating system kernel parsing and processing tasks, identifying attention layers and registering them as kernel-level attention computing primitives; creating attention contexts based on attention layers as management units for resource scheduling and data distribution, encapsulating queries, indexes, content data, and positional dependencies; based on this context, the kernel divides sequence positions into logical attention domains and distributes indexes and content data to corresponding computing nodes; when query data arrives, the kernel calls the primitive to execute distributed attention computing and obtain the dot product result. This invention effectively reduces communication latency in attention computing in a distributed environment, reduces idle waiting time of computing nodes, and improves memory utilization efficiency, thereby improving the overall training and inference performance of the Transformer model in distributed systems.

Description

Technical Field

[0001] This invention belongs to the technical field of computing architecture, specifically relating to a distributed computing scheduling method and management system oriented towards attention models. Background Technology

[0002] In recent years, the Transformer architecture has been widely used in artificial intelligence fields such as natural language processing and computer vision. As the scale of models continues to grow, such as GPT-3 and PaLM, which have hundreds of billions of parameters, a single computing device can no longer meet their training and inference needs. Distributed computing methods must be adopted to deploy computing tasks on multiple computing nodes for parallel execution.

[0003] In the Transformer model, the attention layer is its basic building block. During computation, the attention layer transforms input data into query data, index data, and content data. It then determines attention weights by matching the query and index data, and finally aggregates the content data based on these weights to obtain the output of the current layer. However, as the sequence length increases, the computational and data volume of this process increases quadratically, severely hindering the development of distributed training and inference.

[0004] Currently, some related technologies employ data parallelism, where each computing node holds a complete copy of the model and processes different data shards, updating parameters synchronously through gradients. However, due to the large amount of gradient communication and the need for each node to maintain a complete KV cache, memory redundancy occurs. Other technologies use model parallelism, which splits the attention layer parameters by header or column to different nodes. However, QK^T computation requires all-to-all global communication, with communication volume proportional to the square of the sequence length, and static partitioning leads to poor data locality. In summary, regardless of the technology used, attention computation is treated as a general matrix operation, using general primitives such as all-to-all, resulting in high communication complexity, which increases quadratically with increasing sequence length. Static data partitioning strategies lead to a large number of remote data accesses, placing significant pressure on the device. Using key and value as intermediate data easily leads to redundant I / O and memory copying. Summary of the Invention

[0005] The purpose of this invention is to propose a distributed attention computing method and operating system based on an attention model to solve the problems in the prior art.

[0006] Therefore, the present invention provides a distributed computing scheduling method oriented towards attention models, comprising: In response to the received processing task of the attention model, the operating system kernel parses the processing task to identify the attention layer and registers the computation task corresponding to the attention layer as an operating system kernel-level attention computation primitive; Based on the attention layer, a corresponding attention context is created in the operating system kernel. The attention context is the management unit for resource scheduling and data distribution of the operating system, and the attention context at least encapsulates the query data, index data, content data, and sequence position dependent data of the current attention calculation layer. Based on the attention context, the operating system kernel divides the sequence position into at least one logical attention domain, and distributes the index data and content data to the computing nodes corresponding to each logical attention domain according to the division of the logical attention domain. The computing nodes are physical cores or virtual cores managed by the operating system. When the query data reaches the computing node, the operating system kernel calls the attention computing primitive to perform distributed attention computing on the corresponding computing node to obtain the dot product calculation result of the current attention computing layer.

[0007] In some embodiments, obtaining the dot product calculation result of the current attention calculation layer includes: On the computing node, at least intermediate results of local data are computed, while asynchronously initiating requests for remote data stored on other computing nodes; and The system receives the remote data and aggregates the calculation results based on the remote data with the intermediate results of the local data.

[0008] In some embodiments, the attention context is further used for: The operating system kernel can pre-allocate computing resources and communication bandwidth to the attention layer based on the resource requirement information encapsulated in the attention context, and schedule the computing tasks of the attention layer to suitable computing nodes.

[0009] In some embodiments, distributing the index data and content data of the processing task to the computing nodes corresponding to each logical attention domain includes: The index data and content data at the same sequence position are bound together and mapped to the same computing node, so that when accessing the corresponding content data according to the matching result of the index data in the subsequent attention calculation process, the content data can be obtained locally on the same computing node, thereby reducing cross-node communication overhead and realizing localized data access.

[0010] In some embodiments, the method further includes: performing lifecycle management on the index data and the content data as transient intermediate data, wherein the lifecycle management includes: During the process of asynchronously initiating a request for remote index data and obtaining the complete dot product calculation result, the generated index data and content data are directly routed to the target computing node in a zero-copy manner, without needing to be written to global storage; Once the attention calculation is complete, immediately release the memory occupied by the index data and the content data.

[0011] In some embodiments, releasing the memory occupied by the index data and the content data includes: The index data and content data in the cache are replaced based on the least recently used algorithm or the access frequency algorithm.

[0012] In some embodiments, the lifecycle management further includes: Monitor the access frequency of each computing node to the index data and the content data; When the access frequency of any index data or content data exceeds the threshold, it is dynamically migrated to the computing node that initiated the query.

[0013] In some embodiments, dividing the sequence positions into at least one logical attention domain includes: The size of the logical attention domain is dynamically adjusted based on hardware topology features and / or model parameters.

[0014] In some embodiments, distributing index data and content data to the computing nodes corresponding to each logical attention domain includes: Index data is stored contiguously in the memory of the corresponding compute nodes along the column dimension to ensure the continuity of memory access. Double alignment is applied to the content data.

[0015] In some embodiments, the dual alignment includes outer alignment and inner alignment: The outer alignment involves dividing the content data into multiple sequence blocks according to the sequence position, with each sequence block corresponding to a logical attention domain, and distributing different sequence blocks to different computing nodes. The inner alignment involves caching the content data in the received sequence blocks within each computing node according to the granularity of the attention head, and establishing an independent cache index for each attention head, so that the content data accessed by a single attention head during computation is stored continuously in memory.

[0016] In some embodiments, the asynchronous initiation of a request for remote data stored on other computing nodes includes: Based on the sequence position of the query data, and according to historical access patterns or linear relationships of sequence positions, predictively prefetch the remote index data and content data required for subsequent calculations into the local cache.

[0017] In some embodiments, the predictive prefetching is based on any of the following algorithms: Markov prediction model based on historical access patterns; Linear prediction models based on sequence position; and Intelligent prediction models based on machine learning.

[0018] In some embodiments, the asynchronous initiation of a request for remote index data includes: The request is initiated through a dedicated attention communication protocol, which predicts the contribution of index data in attention calculation, requests only a subset of index data whose contribution is higher than a preset threshold, and selects a transmission path according to the network topology.

[0019] In some embodiments, the aggregation of the calculation results based on the remote data and the intermediate results based on the local data includes: When the accumulated dot product calculation results reach a preset threshold, normalization processing is triggered. or, Based on historical calculation data, predict the contribution of remote data and prioritize aggregating the calculation results of remote index data with high contributions; or, Ignore the dot product calculation results corresponding to remote index data whose attention weight is lower than the preset sparsity threshold.

[0020] In some embodiments, the attention context further encapsulates at least one of the following: The dimension information corresponding to the query data, index data, and content data; Attention head allocation information; and Calculate lifecycle and resource requirements information.

[0021] On the other hand, a distributed computing management system oriented towards attention models is also provided, including: Includes an operating system kernel, wherein the operating system kernel includes: The attention context manager, in response to the received processing task of the attention model, parses the processing task and identifies the attention layer, and registers the computation task corresponding to the attention layer as an operating system kernel-level attention computation primitive; it creates a corresponding attention context according to the attention layer, the attention context being the management unit for resource scheduling and data distribution of the operating system, and the attention context encapsulates at least the query data, index data, content data, and sequence position dependency data of the current attention computation layer; A KV computation execution engine is used to divide sequence positions into at least one logical attention domain based on the attention context, and distribute index data and content data to the computation nodes corresponding to each logical attention domain according to the division of the logical attention domains. The computation nodes are physical cores or virtual cores managed by the operating system. When the query data reaches the computing node, the operating system kernel calls the attention computing primitive to perform distributed attention computing on the corresponding computing node to obtain the dot product calculation result of the current attention computing layer.

[0022] A transient key-value storage system is used to directly route the generated index data and content data to the target computing node during the execution of the attention computation primitive, release the memory it occupies immediately after the attention computation is completed, and dynamically migrate hot data according to the access frequency.

[0023] Beneficial effects: 1. This invention binds and maps index data and content data at the same sequence position to the same computing node, eliminating the need for cross-node retrieval when accessing content data during attention computation. Furthermore, the contiguous column-wise storage of index data ensures continuous memory access, while double alignment of content data ensures that the data accessed by the attention head is continuous in memory, thereby reducing cross-node communication and improving memory access efficiency.

[0024] 2. This invention asynchronously initiates remote data requests while calculating the local dot product, and predictively prefetches the data needed later. After the remote data arrives, it incrementally aggregates into the local result, enabling communication and computation to be executed in parallel, reducing the idle waiting time of the computing nodes.

[0025] 3. This invention routes index data and content data directly to the target computing node as transient data, and releases memory immediately after the calculation is completed, eliminating redundant I / O and memory copying. Furthermore, through cache replacement and dynamic migration, high-frequency access data is migrated to the query node, reducing memory usage and remote access latency. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the distributed attention calculation method of the present invention.

[0028] Figure 2 This is a flowchart illustrating the execution of the distributed attention computation method of the present invention.

[0029] Figure 3 This is a distribution diagram of attention perception data for the present invention.

[0030] Figure 4 This is a flowchart of the communication-computing fusion execution process of the present invention.

[0031] Figure 5 This is a transient KV lifecycle management diagram of the present invention.

[0032] Figure 6 This is a system architecture diagram of the distributed operating system of the present invention. Detailed Implementation

[0033] The invention will be more readily understood by referring to the following detailed description of preferred embodiments and included examples. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In case of conflict, the definitions in this specification shall prevail.

[0034] This invention provides a distributed computing scheduling method and management system oriented towards attention models, solving the problems of excessive communication overhead due to numerous communication cycles and large data volumes, long waiting times due to the serial nature of the computation and data acquisition processes, and low efficiency and large memory consumption in the management of key-value data in existing technologies. The technical concept of this invention lies in elevating attention computation to a kernel-level primitive through the operating system kernel, dividing sequence positions into logical attention domains, and mapping index data and content data at the same sequence position to the same computing node. This allows most data access to be completed locally or on neighboring nodes, reducing the amount of data communicated across nodes. Addressing the problem of long waiting times caused by the serial execution of computation and data acquisition processes in existing solutions, this invention asynchronously requests remote data while calculating the local dot product, and aggregates the incremental remote results into the local results, enabling the communication and computation processes to overlap. Addressing the issues of low efficiency and high memory consumption in key-value data management in existing solutions, this invention manages index data and content data as transient data with a defined lifecycle. After generation, they are directly routed to the target computing node, and memory is released immediately after computation. This effectively improves the efficiency of attention computation in a distributed environment.

[0035] Accordingly, this invention provides a distributed operating system, AttentOS, which includes an operating kernel. The operating kernel comprises three core components: an attention context manager, a key-value (KV) computation execution engine, and a transient KV storage system. The attention context manager is responsible for identifying semantic information in the attention computation layer, providing a basis for subsequent optimization. The KV computation execution engine is responsible for implementing specific operations such as logical attention domain partitioning, local dot product calculation, remote data requests, and incremental aggregation. The transient KV storage system is responsible for the routing and lifecycle management of index data and content data. These three core components work together to optimize the efficiency of distributed attention computation.

[0036] Please refer to the following examples for details: like Figure 1-5 As shown, a distributed computing scheduling method oriented towards an attention model, executed by the operating system kernel, includes: In S100, in response to the received processing task of the attention model, the operating system kernel parses the processing task to identify the attention layer and registers the computation task corresponding to the attention layer as an operating system kernel-level attention computation primitive. This task can be a training or inference task built with Transformer, or a task built with other attention-type models (such as Perceiver, Linear Transformer), or a non-Transformer architecture model that includes an attention mechanism.

[0037] In step S200, based on the attention layer, a corresponding attention context is created in the operating system kernel. The attention context serves as the management unit for resource scheduling and data distribution within the operating system. This attention context encapsulates the complete semantics of a single attention computation within the Transformer, including at least the query data, index data, content data, and sequence position dependencies of the current attention computation layer. This step allows the system to obtain basic information about the current computation layer, such as sequence length, data dimensions, and the logical relationships between data points, before performing subsequent data distribution and computation scheduling, providing a basis for subsequent optimization operations.

[0038] When the Transformer task starts, the operating system kernel automatically creates an attention context and dynamically allocates resources based on model parameters such as sequence length and the number of attention heads. Instead of breaking down the computation into general matrix operations, the operating system kernel directly parses the computation graph of the attention mechanism. This transforms resource allocation from a passive response to an active adaptation, effectively improving subsequent computational efficiency.

[0039] It should be noted that the query data is defined as Q, specifically the Query matrix; the index data is defined as K, specifically the Key matrix; and the content data is defined as V, specifically the Value matrix.

[0040] The Transformer is a neural network architecture based on an attention mechanism, widely used in natural language processing, computer vision, and other fields. A Transformer model consists of multiple stacked encoder and decoder layers, each containing an attention layer and a feedforward network layer. The attention layer is the core component of the Transformer; its role is to establish dependencies between different positions in a sequence, enabling each position to obtain information from other positions.

[0041] In S300, based on attention context, the operating system kernel divides sequence positions into at least one logical attention domain. According to this division, index data and content data are distributed to the corresponding computing nodes for each logical attention domain. These computing nodes are physical or virtual cores managed by the operating system. Index data and content data for the same sequence position are mapped to the same computing node. Specifically, the criteria for dividing logical attention domains can include sequence length, the number of computing nodes, and hardware topology. By grouping sequence positions and mapping them to specific computing nodes, each node is primarily responsible for processing sequence positions within a specific range. Most access to index data in subsequent calculations can be completed locally or on neighboring nodes, thereby reducing the amount of data and complexity of cross-node communication.

[0042] S400, when the query data reaches the computing node, the operating system kernel calls the attention computing primitive to perform distributed attention computing on the corresponding computing node to obtain the dot product calculation result of the current attention computing layer. In a preferred embodiment, obtaining the dot product calculation result of the current attention computing layer includes at least computing the intermediate result of the local data on the computing node, while asynchronously initiating a request for remote data stored on other computing nodes; and It receives remote data and aggregates the calculation results based on the remote data with the intermediate results of the local data.

[0043] Specifically, based on the logical attention domain to which the queried data belongs, the local dot product of the local index data is calculated on the corresponding computing node, while simultaneously initiating an asynchronous request for remote index data. After receiving the remote index data, the calculation result of the remote index data is incremented into the calculation result of the local dot product, and this process is repeated until the complete dot product calculation result is obtained. This step changes the originally serial data acquisition and calculation process to parallel execution, enabling computing nodes to continue processing local data while waiting for remote data, thus reducing the idle waiting time of computing nodes.

[0044] In one embodiment, the index data and content data of the processing task are distributed to the computing nodes corresponding to each logical attention domain, including: By binding index data and content data at the same sequence position together and mapping them to the same computing node, the content data can be retrieved locally on the same computing node when accessing the corresponding content data based on the matching result of the index data during subsequent attention calculation. This reduces cross-node communication overhead and enables localized data access.

[0045] Specifically, index data and content data at the same sequence position are bound together to form key-value pairs and stored on the same computing node. Basic unit nodes within the computing node exchange key-value pairs and KV pairs via a dedicated attention-based communication protocol, eliminating the need for all-to-all communication as in existing technologies. When subsequent computations require accessing content data based on the matching results of the index data, both required data are already on the same node, eliminating the need for cross-node retrieval and improving cache hit rate by 70%.

[0046] Then, the dot product calculation result is normalized to obtain the attention weight, and the content data is weighted and aggregated using the attention weight to generate the output of the current attention calculation layer.

[0047] In one embodiment, this embodiment also provides step S500, which employs transient KV lifecycle management. Specifically, the generated index data and content data are directly routed to the target computing node with zero copying, without writing to global storage, and the memory they occupy is released immediately after attention calculation is completed. Specifically, the system treats K and V as transient intermediate data, whose lifecycle is strictly constrained by the attention context. After K and V are generated, the system directly routes them to the attention computing node without writing to global storage. The kernel maintains a KV ready queue; when K... j Upon generation, the computation of dependent Q queries is automatically triggered. After the attention computation is complete, K and V are immediately released to avoid memory fragmentation. This step effectively eliminates the overhead of application-layer management of KV cache, making it more suitable for long sequence inference scenarios and solving the problems of low KV data management efficiency and large memory consumption in existing solutions.

[0048] like Figure 5 As shown, in one embodiment, transient KV lifecycle management is employed. Specifically, the generated index data and content data are directly routed to the target computing node, and the memory they occupy is released immediately after the attention calculation is completed. The system treats K and V as transient intermediate data, whose lifecycle is strictly constrained by the attention context. After K and V are generated, the system directly routes them to the attention computing node without writing them to global storage. The kernel maintains a KV ready queue. j Upon generation, the computation of dependent Q queries is automatically triggered. After the attention computation is complete, K and V are immediately released to avoid memory fragmentation. This step effectively eliminates the overhead of application-layer management of KV cache, making it more suitable for long sequence inference scenarios and solving the problems of low KV data management efficiency and large memory consumption in existing solutions.

[0049] In some feasible embodiments, releasing the memory it occupies involves replacing the index and content data in the cache based on a least recently used algorithm or an access frequency algorithm, enabling compute nodes to efficiently cache hot data with limited memory resources. When using the least recently used algorithm for cache replacement, the system maintains access time records for the index and content data cached on each compute node. Whenever a query accesses a key-value pair, the system updates the most recent access timestamp of that key-value pair.

[0050] When a node runs out of memory and needs to load new index or content data, the system scans all data in the current cache and looks for the least recently used key-value pairs, i.e., the data items with the largest difference between the current timestamp and the most recently accessed timestamp. These data items are considered less likely to be accessed again in the future, so they are preferentially removed from the cache to free up memory space for new data.

[0051] When using a cache replacement algorithm based on access frequency, the system maintains access counts for the index and content data cached on each compute node. Whenever a query accesses a key-value pair, the system increments the access count for that key-value pair.

[0052] When a node's memory is insufficient and new index or content data needs to be loaded, the system compares the access frequency of each data item in the current cache and selects the key-value pair with the lowest access count for eviction. Considering that access patterns may change over time, this method can use a counting method with a decay factor, that is, periodically multiplying the historical access count by a decay coefficient less than 1, so that the recent access frequency has a higher weight in the eviction decision.

[0053] To further optimize data locality, this method also includes an embodiment that dynamically adjusts the data distribution based on access frequency: In one embodiment, transient key-value lifecycle management also includes dynamic key-value migration, specifically monitoring the access frequency of each computing node to index data and content data. When the access frequency of any index data or content data exceeds the threshold, it is dynamically migrated to the computing node that initiated the query.

[0054] By monitoring the access frequency of index and content data across each compute node, frequently accessed hotspot data is identified and dynamically migrated to the vicinity of the compute node initiating the query. After migration, subsequent access to this hotspot data changes from remote access to local or neighboring node access, further reducing cross-node communication and lowering access latency. Compared to static cache replacement strategies, dynamic migration adaptively adjusts data distribution to adapt to changing access patterns, making it particularly suitable for handling long-sequence inference scenarios with uneven query distribution, effectively improving the overall system response speed.

[0055] Based on the above data distribution and calculation process, to further optimize the utilization efficiency of memory resources, this method also includes the following refinement of the data distribution: like Figure 3 As shown, in one embodiment, dividing the sequence position into at least one logical attention domain includes: The size of the logical attention domain is dynamically adjusted based on hardware topology characteristics and / or model parameters.

[0056] Specifically, based on the created attention context, the sequence positions are divided into multiple logical attention domains. Each logical attention domain corresponds to a continuous sequence position, exemplified as LAD. iThe position [i, i + Δ - 1] is included, and its value Δ can be dynamically adjusted based on hardware topology characteristics such as inter-GPU network bandwidth and latency, as well as current model parameters such as the number of attention heads and sequence length. For example, when high inter-node network latency is detected, the system appropriately reduces the value of Δ to restrict more data access to the local machine. When processing very long sequences, the system appropriately increases the value of Δ to reduce the number of cross-domain connections.

[0057] Subsequently, according to the logical attention domain division, the index data and content data are distributed to the corresponding computing nodes of each domain. Two layout strategies are employed during the data distribution process: One approach involves binding index data and content data at the same sequence position to form key-value pairs, which are then stored on the same compute node. Basic unit nodes within the compute node exchange key-value pairs and KV pairs via a dedicated attention-based communication protocol, eliminating the need for all-to-all communication as in existing technologies. When subsequent computations require accessing content data based on the matching results of the index data, both required data are already on the same node, eliminating the need for cross-node retrieval and potentially improving cache hit rate by 70%.

[0058] Another approach is to store index data contiguously in memory along the column dimension. This ensures continuous memory access when performing dot product calculations between index data and query data, improving cache utilization and increasing PCIe transmission efficiency by 2.3 times. It is especially suitable for attention calculations with high bandwidth requirements.

[0059] Based on the completion of logical attention domain division and basic data layout, this method also includes the following embodiments to further optimize the access efficiency of content data: In one embodiment, a dual alignment strategy is implemented for the content data. First, outer alignment is performed, which divides the content data into multiple sequence blocks based on sequence position. The size of each sequence block is consistent with the size Δ of the logical attention domain, and the different sequence blocks are distributed to different computing nodes. Outer alignment ensures that each computing node is responsible for processing the content data corresponding to a consecutive sequence position.

[0060] Within each computing node, inner alignment is performed, which involves further segmenting and caching the content data in the received sequence blocks according to the granularity of the attention heads. Specifically, the system establishes an independent cache area for each attention head, ensuring that the content data accessed by a single attention head during computation is stored contiguously in memory, and maintains an independent cache index for each head to support fast location. When the number of attention heads changes, the system automatically adjusts the cache structure to adapt to the new configuration.

[0061] By using double-aligned caching of content data, the computation of attention heads can access memory continuously, reducing cache misses and improving computational efficiency.

[0062] like Figure 4 As shown in one embodiment, the method for overlapping the communication and computation processes during dot product calculation to reduce the idle waiting time of the computation node is explained, specifically: Calculating the local dot product, i.e., calculating QK T Simultaneously, based on the sequence position of the query data, and according to historical access patterns or linear relationships of sequence positions, the system predictively prefetches the remote index data and content data required for subsequent computations into the local cache. When the query data arrives at the computation node, the system predicts the remote index data and content data that may be needed for subsequent computations based on the position information of the query data in the sequence. Specifically, the system maintains access pattern statistics between each logical attention domain. When processing query data at the i-th position, it predicts the range of remote sequence positions that the position may depend on based on historical access records and initiates a data prefetching request to the corresponding remote node in advance.

[0063] This prefetch operation is executed by kernel events. When a query data ready event is triggered, the system automatically schedules the corresponding prefetch instruction, allowing the data request and local computation to start simultaneously. In some embodiments, the prefetch operation can employ any of the following linear relationships of sequence positions, i.e., algorithms: Markov prediction model based on historical access patterns; Linear prediction models based on sequence position; and Intelligent prediction models based on machine learning. These can be selected according to actual needs.

[0064] Then, local dot product calculations and asynchronous requests are performed. On the current node, the system immediately begins calculating the local dot product using local index data to obtain the dot product result within the logical attention domain of the current node. Simultaneously, the system asynchronously sends data requests to the node storing remote index data, requesting the required subset of index data.

[0065] Since the index data and content data are bound and stored on the same node, when the remote index data arrives, its corresponding content data is already ready on the same node, so there is no need to request it again.

[0066] Building upon predictive prefetching and asynchronous requests, this method also includes optimizations to the communication protocol to further reduce data transmission volume: In one embodiment, when initiating a remote data request, an attention-specific communication protocol is used instead of general all-to-all communication. This protocol requests only the subset of index data needed for computation and selects a transmission path based on the network topology.

[0067] After the remote data arrives, this method incorporates the data into the calculation results through incremental aggregation, specifically including the following implementation methods: Incremental aggregation is performed whereby, as remote index data arrives at the current node, the system immediately and incrementally aggregates the resulting dot product into the previously obtained local dot product. The aggregation operation is streamed, with remote data arriving and being aggregated batch by batch, without waiting for all remote data to arrive before starting the calculation.

[0068] Throughout the aggregation process, local computation continues, and new query data can continue to enter the processing pipeline, forming a multi-stage pipeline in parallel.

[0069] Among them, aggregating the incremental dot product calculation results generated from remote index data into the local dot product calculation results includes any of the following methods: When the accumulated dot product calculation results reach a preset threshold, subsequent calculations are triggered; the system sets a preset threshold for the dot product calculation results. As remote index data arrives in batches, the system continuously adds the resulting dot product calculation results to the local result, while monitoring the completion rate of the current accumulated value. When the accumulated value reaches the preset threshold, the system determines that the current result has sufficient accuracy, thus triggering the subsequent normalization processing steps without waiting for all remote data to arrive.

[0070] When initiating remote data requests, the attention-specific communication protocol also judges the contribution of index data in attention computation, requesting only a subset of index data whose contribution exceeds a preset threshold, and selecting a transmission path based on network topology. Specifically, it predicts the contribution of remote data based on historical computation data and selectively aggregates it; the system pre-collects historical computation data and establishes a statistical prediction model for the contribution of remote data. For the current computation task, before initiating a remote data request, the system predicts the contribution of each remote data block to the final dot product calculation result based on the location information and historical statistical patterns of the queried data. Based on the prediction results, the system sorts or filters remote data requests. For remote data with a high predicted contribution, requests are initiated and aggregation is performed first; for data with a low predicted contribution, requests can be delayed or their priority reduced. During the aggregation process, the system dynamically adjusts the subsequent request strategy based on the contribution of the actual arriving data.

[0071] The system ignores the dot product calculation results corresponding to remote index data whose attention weights are lower than a preset sparsity threshold. When processing remote index data, the system pre-estimates or quickly determines whether the attention weights corresponding to each remote data point might be lower than the preset sparsity threshold. For data whose weights might be lower than the threshold, the system selectively ignores their corresponding dot product calculation results, does not perform aggregation operations, or only performs low-precision approximate aggregation.

[0072] After completing the calculation and generating the output, this method also includes the reclamation and management of memory resources, specifically including the following embodiments.

[0073] Figure 6 As shown, this invention also provides a distributed computing management system for attention models, including an operation kernel. The operation kernel comprises three core components: an attention context manager, a key-value (KV) computation execution engine, and a transient key-value (KV) storage system. These three components work together to optimize the efficiency of distributed attention computation. Wherein: Explanation of the attention context manager The attention context manager is responsible for parsing and processing tasks and identifying attention layers after the system receives training or inference tasks. It then registers the computation tasks corresponding to these attention layers as kernel-level attention computation primitives. Based on the attention layer, it creates a corresponding attention context, which serves as the operating system's management unit for resource scheduling and data distribution. When a Transformer task is submitted to the system, the attention context manager first parses the model structure involved in the task, identifying the layers that require attention computation. For each attention computation layer, the manager creates an independent attention context.

[0074] This attention context encapsulates the complete semantic information of the current attention computation layer, including at least: query data, index data, content data and their logical relationships; dependencies between sequence positions; dimensional information corresponding to query data, index data, and content data; attention head allocation information; and computation lifecycle and resource requirements information. This information provides a basis for subsequent data distribution, computation scheduling, and resource management.

[0075] The attention context manager pre-allocates computing resources and communication bandwidth based on the resource requirement information encapsulated in the context, and schedules tasks to suitable computing nodes. In this way, the system can learn about the characteristics of the current computing layer before executing specific calculations, transforming resource allocation from a passive response to an active adaptation.

[0076] The KV calculation execution engine is explained below: The KV computation execution engine is the core computational unit of the system, responsible for implementing the specific distributed attention computation process. Based on the attention context created by the attention context manager, this engine performs the following operations: First, the KV computation execution engine divides the sequence position into at least one logical attention domain based on the sequence length, the number of computation nodes, and hardware topology characteristics. Each logical attention domain corresponds to a continuous sequence position, and its size can be dynamically adjusted according to hardware characteristics such as network latency and bandwidth, as well as model parameters. After the division, the engine distributes the index data and content data to the computation nodes corresponding to each domain according to the logical attention domain division. The computation nodes are physical cores or virtual cores managed by the operating system. During the data distribution process, the index data and content data at the same sequence position are bound together to form key-value pairs and co-located on the same computation node; simultaneously, the index data is stored contiguously in memory along the column dimension to ensure the continuity of memory access.

[0077] For content data, the KV computation execution engine further implements a dual alignment strategy. The outer alignment divides the content data into multiple sequence blocks based on sequence position, with each sequence block being the same size as the logical attention domain, and distributes these different sequence blocks across different computation nodes. The inner alignment, within each computation node, caches the content data in the received sequence blocks at the attention head granularity, and establishes an independent cache index for each attention head, ensuring that the content data accessed by a single attention head during computation is stored contiguously in memory.

[0078] When query data arrives at the compute node, the operating system kernel invokes attention computation primitives to perform distributed attention computation on the corresponding compute node to obtain the dot product calculation result of the current attention computation layer. On the current node, the engine immediately uses local index data to calculate the local dot product, obtaining the dot product calculation result within the logical attention domain of the current node. Simultaneously, the engine asynchronously sends data requests to nodes storing remote index data, requesting the required subset of index data. When initiating remote requests, the engine employs a dedicated attention communication protocol, which only requests the necessary subset of index data for computation and selects the transmission path based on the network topology.

[0079] While waiting for remote data, local computation continues. When remote index data arrives at the current node, the engine immediately incrementally aggregates its generated dot product results into the previously obtained local dot product results. Incremental aggregation can take several forms, such as: triggering subsequent computations when the accumulated dot product results reach a preset threshold; selectively aggregating based on the predicted contribution of remote data according to historical computation data; or ignoring dot product results corresponding to remote index data with attention weights below a preset sparsity threshold.

[0080] Once all remote data has been processed, the engine obtains the complete dot product calculation result. This result is then normalized to obtain attention weights, which are used to weight and aggregate the corresponding content data, generating the output of the current attention calculation layer.

[0081] Explanation of transient key-value storage systems: The transient key-value storage system is responsible for the lifecycle management of index data and content data. This system treats index data and content data as transient intermediate data with a defined lifecycle, whose lifecycle is strictly constrained by the attention context.

[0082] After the previous layer calculates and generates index and content data, the transient key-value storage system directly routes the generated index and content data to the target compute node without writing to global storage. The system kernel maintains a key-value ready queue, automatically triggering query computations that depend on new key-value pairs when they are generated.

[0083] During the attention calculation process, the transient key-value storage system monitors the access frequency of index data and content data on each computing node. When the access frequency of any index data or content data exceeds a preset threshold, the system dynamically migrates it to the computing node that initiated the query. Based on the access frequency, the system dynamically migrates hot data, so that subsequent access to the hot data changes from remote access to local or neighboring node access.

[0084] Once the attention calculation is complete, the transient key-value storage system immediately releases the memory occupied by the index and content data. When memory resources are limited, the system employs a cache replacement strategy to manage memory space. For example, based on the least recently used algorithm or access frequency algorithm, the system replaces the index and content data in the cache to ensure that frequently accessed data can be retained in memory.

[0085] It should be noted that the distributed operating system provided in this application, namely AttentOS, can be implemented at different levels, such as as an independent operating system, as an extension module of an existing operating system, or as firmware for a hardware accelerator, as long as it can provide functionality.

[0086] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed computing scheduling method oriented towards attention models, characterized in that, include: In response to the received processing task of the attention model, the operating system kernel parses the processing task to identify the attention layer and registers the computation task corresponding to the attention layer as an operating system kernel-level attention computation primitive; Based on the attention layer, a corresponding attention context is created in the operating system kernel. The attention context is the management unit for resource scheduling and data distribution of the operating system, and the attention context at least encapsulates the query data, index data, content data, and sequence position dependent data of the current attention calculation layer. Based on the attention context, the operating system kernel divides the sequence position into at least one logical attention domain, and distributes the index data and content data to the computing nodes corresponding to each logical attention domain according to the division of the logical attention domain. The computing nodes are physical cores or virtual cores managed by the operating system. When the query data reaches the computing node, the operating system kernel calls the attention computing primitive to perform distributed attention computing on the corresponding computing node to obtain the dot product calculation result of the current attention computing layer.

2. The method according to claim 1, characterized in that, The process of obtaining the dot product calculation result of the current attention calculation layer includes: On the computing node, at least intermediate results of local data are computed, while asynchronously initiating requests for remote data stored on other computing nodes; and The system receives the remote data and aggregates the calculation results based on the remote data with the intermediate results of the local data.

3. The method according to claim 1, characterized in that, The attention context is also used for: The operating system kernel can pre-allocate computing resources and communication bandwidth to the attention layer based on the resource requirement information encapsulated in the attention context, and schedule the computing tasks of the attention layer to suitable computing nodes.

4. The method according to claim 1, characterized in that, The step of distributing the index data and content data of the processing task to the computing nodes corresponding to each logical attention domain includes: The index data and content data at the same sequence position are bound together and mapped to the same computing node, so that when accessing the corresponding content data according to the matching result of the index data in the subsequent attention calculation process, the content data can be obtained locally on the same computing node, thereby reducing cross-node communication overhead and realizing localized data access.

5. The method according to claim 1, characterized in that, Also includes: The index data and the content data are treated as transient intermediate data and managed for lifecycle purposes. This lifecycle management includes: During the process of asynchronously initiating a request for remote index data and obtaining the complete dot product calculation result, the generated index data and content data are directly routed to the target computing node in a zero-copy manner, without needing to be written to global storage; Once the attention calculation is complete, immediately release the memory occupied by the index data and the content data.

6. The method according to claim 5, characterized in that, Releasing the memory occupied by the index data and the content data includes: The index data and content data in the cache are replaced based on the least recently used algorithm or the access frequency algorithm.

7. The method according to claim 5, characterized in that, The lifecycle management also includes: Monitor the access frequency of each computing node to the index data and the content data; When the access frequency of any index data or content data exceeds the threshold, it is dynamically migrated to the computing node that initiated the query.

8. The method according to claim 1, characterized in that, The step of dividing the sequence position into at least one logical attention domain includes: The size of the logical attention domain is dynamically adjusted based on hardware topology features and / or model parameters.

9. The method according to claim 1, characterized in that, The process of distributing index data and content data to the computing nodes corresponding to each logical attention domain includes: Index data is stored contiguously in the memory of the corresponding compute nodes along the column dimension to ensure the continuity of memory access. Double alignment is applied to the content data.

10. The method according to claim 9, characterized in that, The dual alignment includes outer alignment and inner alignment: The outer alignment involves dividing the content data into multiple sequence blocks according to the sequence position, with each sequence block corresponding to a logical attention domain, and distributing different sequence blocks to different computing nodes. The inner alignment involves caching the content data in the received sequence blocks within each computing node according to the granularity of the attention head, and establishing an independent cache index for each attention head, so that the content data accessed by a single attention head during computation is stored continuously in memory.

11. The method according to claim 2, characterized in that, The asynchronous initiation of requests for remote data stored on other computing nodes includes: Based on the sequence position of the query data, and according to historical access patterns or linear relationships of sequence positions, predictively prefetch the remote index data and content data required for subsequent calculations into the local cache.

12. The method according to claim 11, characterized in that, The predictive prefetching is based on any of the following algorithms: Markov prediction model based on historical access patterns; Linear prediction models based on sequence position; and Intelligent prediction models based on machine learning.

13. The method according to claim 2, characterized in that, The asynchronous initiation of a request for remote index data includes: The request is initiated through a dedicated attention communication protocol, which predicts the contribution of index data in attention calculation, requests only a subset of index data whose contribution is higher than a preset threshold, and selects a transmission path according to the network topology.

14. The method according to claim 2, characterized in that, The aggregation of the calculation results based on the remote data and the intermediate results based on the local data includes: When the accumulated dot product calculation results reach a preset threshold, normalization processing is triggered. or, Based on historical calculation data, predict the contribution of remote data and prioritize aggregating the calculation results of remote index data with high contributions; or, Ignore the dot product calculation results corresponding to remote index data whose attention weight is lower than the preset sparsity threshold.

15. The method according to claim 1, characterized in that, The attention context also encapsulates at least one of the following: The dimension information corresponding to the query data, index data, and content data; Attention head allocation information; and Calculate lifecycle and resource requirements information.

16. A distributed computing management system oriented towards an attention model, characterized in that, Includes an operating system kernel, wherein the operating system kernel includes: The attention context manager, in response to the received processing task of the attention model, parses the processing task and identifies the attention layer, and registers the computation task corresponding to the attention layer as an operating system kernel-level attention computation primitive; it creates a corresponding attention context according to the attention layer, the attention context being the management unit for resource scheduling and data distribution of the operating system, and the attention context encapsulates at least the query data, index data, content data, and sequence position dependency data of the current attention computation layer; A KV computation execution engine is used to divide sequence positions into at least one logical attention domain based on the attention context, and distribute index data and content data to the computation nodes corresponding to each logical attention domain according to the division of the logical attention domains. The computation nodes are physical cores or virtual cores managed by the operating system. When the query data reaches the computing node, the operating system kernel calls the attention computing primitive to perform distributed attention computing on the corresponding computing node to obtain the dot product calculation result of the current attention computing layer. A transient key-value storage system is used to directly route the generated index data and content data to the target computing node during the execution of the attention computation primitive, release the memory it occupies immediately after the attention computation is completed, and dynamically migrate hot data according to the access frequency.

Images