A data access system and method

Abstract

The application provides a data access system and method, relates to the technical field of data management, and adopts a last-stage controller to manage cache consistency in an accelerator and an upper-stage controller to manage cache consistency between accelerators, each controller only needs to maintain information in a limited range, avoids the problem of large management overhead caused by centralized management, and improves system scalability. The last-stage controller directly determines local or remote types according to an address mapping relationship, shortens the determination delay. Local access directly reads local memory, and the time delay is the lowest. Remote access is preferentially obtained from other acceleration sub-modules that have cached the data in the accelerator, so that the cross-accelerator memory access and communication pressure are reduced. The last-stage controller only needs to send a consistency message to the upper-stage controller once, the upper-stage controller uniformly forwards the consistency message to other related last-stage controllers to complete cache state adjustment, reduces the maintenance overhead, and guarantees data consistency. On the premise of guaranteeing consistency of a large-scale accelerator cluster, the data access efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a data access system and method. Background Technology

[0002] The current demand for large-scale computing has driven the deployment of accelerator clusters, with the number of processor cores within these clusters now reaching hundreds or even thousands. Each processor core typically has a private cache and shares local memory. To reduce data transfer overhead, cache consistency needs to be maintained both within and between accelerators. However, traditional snooping protocols are prone to broadcast storms as the number of processor cores increases, resulting in poor scalability. Traditional directory protocols record sharer information that grows quadratically with the number of processor cores, leading to enormous storage and query overhead. While two-level directory schemes proposed in related technologies allow for hierarchical querying based on the cluster master node and the master node within each accelerator, directory entries still need to track sharer information related to the total number of devices. As the accelerator scales up, routing and directory overhead remains substantial, making it difficult to support the efficient expansion of large-scale accelerator clusters. Therefore, improving the efficiency of data access within the cluster is a problem that those skilled in the art need to solve. Summary of the Invention

[0003] This application provides a data access system and method to at least address the problem of low access efficiency for data within a cluster in related technologies.

[0004] This application provides a data access system, including multiple accelerators, a final-level controller deployed inside the multiple accelerators, and at least one upper-level controller; wherein, the upper-level controller is used to connect to the multiple final-level controllers; each accelerator includes multiple acceleration sub-modules; each acceleration sub-module has its corresponding cache and local memory; The acceleration submodule is connected to the final controller deployed inside its accelerator, and is used to send the first memory access request to the final controller and receive access data fed back by the final controller. The final-level controller receives the first memory access request sent by the acceleration submodule, determines the address type of the access address carried in the first memory access request based on the address mapping relationship; wherein the address type includes local address and remote address; if the access address belongs to local address, it obtains access data from local memory; if the access address belongs to remote address, it obtains access data from other acceleration submodules that match the access address; it feeds back the access data to the acceleration submodule and sends a consistency message to the upper-level controller to which it is connected; The upper-level controller is used to forward consistency messages to other cached data access controllers, so that the other cached data access controllers can adjust the cache state of the accessed data.

[0005] This application also provides a data access method applicable to a final-level controller, which is deployed within an accelerator and connected to multiple acceleration submodules within the accelerator; the method includes: The system receives a first memory access request sent by the acceleration submodule and determines the address type of the access address carried in the first memory access request based on the address mapping relationship; wherein, the address type includes local address and remote address; If the access address is a local address, the accessed data is retrieved from local memory; If the access address is a remote address, obtain access data from other acceleration submodules that match the access address; The access data is fed back to the acceleration submodule and a consistency message is sent to the upper controller it is connected to, so that the upper controller can forward the consistency message to the other end controllers that cache the access data, so that the other end controllers that cache the access data can adjust the cache state of the access data.

[0006] This application employs a method where a final-level controller manages the cache consistency within the accelerator, while a higher-level controller manages the cache consistency between accelerators. Each final-level controller only needs to maintain information on a limited number of acceleration submodules within its own accelerator, and the higher-level controller only needs to manage a limited number of final-level controllers connected to it. Compared to a single control node centrally managing all acceleration submodules, this avoids the problem of the management information volume expanding rapidly with the total number of accelerators and acceleration submodules, thus giving the system better scalability and enabling it to support larger-scale accelerator clusters. Acceleration submodules only need to send access requests to the final-level controller of their respective accelerator, without needing to determine the data storage location themselves, simplifying the design complexity of the acceleration submodules. The final-level controller directly determines the local or remote type of the access address based on the address mapping relationship, eliminating the need for repeated global traversal searches and reducing the latency required for address determination. When the access address is a local address, data is directly retrieved from local memory, resulting in the shortest access path and lowest latency. When accessing a remote address, data is preferentially retrieved from other acceleration submodules within the same accelerator that match that address, rather than directly accessing remote memory across accelerators. This fully utilizes existing data replicas, reduces unnecessary cross-accelerator memory accesses, and lowers communication pressure between accelerators. When the last-level controller reports accessed data, it only needs to send a consistency message once to its connected upper-level controller. The upper-level controller then forwards this message to other last-level controllers that cache the data, prompting them to adjust their cache states. This eliminates the need for the last-level controllers to notify each node in the cluster that may hold a copy of the data individually. This reduces the communication overhead required for consistency maintenance and ensures that the data used by each acceleration submodule in the cluster is always in a consistent state, avoiding the use of expired data replicas. While ensuring cache consistency in a large-scale accelerator cluster, this effectively reduces the storage and communication overhead of consistency management, shortens data access paths, and thus improves data access efficiency. Attached Figure Description

[0007] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram of the structure of a data access system provided in an embodiment of this application; Figure 2 A schematic diagram illustrating a local memory access method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating remote access with access data present on a local accelerator, provided as an embodiment of this application; Figure 4 A schematic diagram illustrating a remote access scenario where the local accelerator does not have access data, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of a structure for interconnecting multiple GPUs, provided in an embodiment of this application. Figure 6 A schematic diagram of a layered topology interconnection of a data access system provided in this application embodiment; Figure 7 A flowchart illustrating a data access method applied to a final-level controller, as provided in an embodiment of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0010] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0011] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] Example 1: Figure 1 The present application provides a schematic diagram of the structure of a data access system, including multiple accelerators 10, a final-level controller 20 deployed inside the multiple accelerators 10, and at least one upper-level controller 30; wherein the upper-level controller 30 is used to connect to the multiple final-level controllers 20; each accelerator 10 includes multiple acceleration sub-modules 101; each acceleration sub-module 101 has its corresponding cache and local memory.

[0013] Accelerator 10 can be a processing device with massively parallel computing capabilities, such as a graphics processing unit (GPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In this application embodiment, a GPU is used as an example for description.

[0014] The acceleration submodule 101 can be understood as a relatively independent computing module within the accelerator. Taking the accelerator 10 using a GPU as an example, the acceleration submodule 101 can be the graphics processing module (GPM) in the GPU. Each acceleration submodule 101 integrates a processor core such as a streaming multiprocessor (SM), a private cache (L1), and a last-level cache (L2), and is equipped with local memory (DRAM). The private cache (L1) and the last-level cache (L2) can be collectively referred to as cache.

[0015] It is becoming increasingly difficult to further increase the integration of a single chip. Breaking down an accelerator into multiple accelerator sub-modules that can be packaged within the same accelerator can increase the computing power density of a single accelerator under limited process conditions, and also facilitates the subsequent construction of larger-scale computing clusters.

[0016] The acceleration submodule 101 is connected to the final controller 20 deployed inside its accelerator 10, and is used to send a first memory access request to the final controller 20 and receive access data fed back by the final controller 20.

[0017] The final controller 20 is deployed inside each accelerator 10 to manage cache consistency among the various acceleration submodules 101 within the accelerator 10.

[0018] Taking an acceleration submodule 101 as an example, in this embodiment of the application, in order to facilitate the distinction between different accelerators, the accelerator to which the acceleration submodule 101 belongs can be called the local accelerator, and other accelerators can be called remote accelerators.

[0019] The functions of the upper-level controller 30 and the lower-level controller 20 are similar, the difference being that they manage different objects. The upper-level controller 30 is used to connect multiple lower-level controllers 20 and manage the cache consistency among the accelerators 10.

[0020] Taking accelerator 10, which uses a GPU, as an example, the final controller can be a GPU controller (GC). The upper-level controller can be a GC controller (GCC).

[0021] Considering that if a single-level centralized management of the cache state of all acceleration submodules is adopted, the amount of information that the centralized directory needs to record will increase dramatically as the number of accelerators and acceleration submodules grows, leading to a decrease in query and maintenance efficiency. Therefore, this application sets up a multi-level management approach where the last-level controller manages the internal accelerators and the upper-level controller manages the inter-accelerators. By splitting consistency management into two levels—internal and inter-accelerator—the last-level controller only needs to manage a limited number of acceleration submodules within the accelerator, and the upper-level controller only needs to manage a relatively limited number of last-level controllers. This decomposes the originally massive centralized consistency management problem into several smaller sub-problems and facilitates subsequent expansion of the system by increasing the number of controllers.

[0022] The final stage controller 20 is used to receive the first memory access request sent by the acceleration submodule 101 and determine the address type of the access address carried by the first memory access request according to the address mapping relationship; wherein, the address type may include local address and remote address.

[0023] The address mapping relationship can be a pre-defined correspondence between the access address and its associated acceleration submodule 101 or accelerator 10. Through this mapping relationship, the final controller 20 can quickly determine whether the data corresponding to the current access address should be obtained from local memory or from other acceleration submodules 101, without having to perform a global traversal search for each access, thus improving the efficiency of address determination.

[0024] If the access address is a local address, the accessed data is retrieved from local memory.

[0025] If the access address is a remote address, the access data is obtained from other acceleration submodules 101 that match the access address.

[0026] After obtaining the access data, the final controller 20 feeds the access data back to the acceleration submodule 101 and sends a consistency message to the upper controller 30 to which it is connected.

[0027] If the accessed address is a local address, it means that the required data is stored in local memory. In this case, the data can be retrieved directly from local memory. Directly accessing local memory has the shortest path and the lowest latency.

[0028] If the access address is a remote address, the system first checks whether the caches of other acceleration submodules 101 within the local accelerator have recorded the required access data. If the caches of other acceleration submodules 101 within the local accelerator have not recorded the required access data, then cross-accelerator 10 access is performed.

[0029] In this application, data is preferentially obtained from other acceleration submodules within the local accelerator instead of directly accessing remote memory across accelerators. This allows full utilization of the data copies recorded in the cache of each acceleration submodule 101, reducing unnecessary cross-accelerator accesses.

[0030] After obtaining the accessed data, the last-level controller 20 sends a consistency message to the upper-level controller 30, thereby notifying other last-level controllers 20 in the cluster that may have cached the accessed data to synchronously adjust the cache state, ensuring the consistency of the same data in different storage locations.

[0031] The upper-level controller 30 is used to forward consistency messages to the lower-level controller 20 that accesses other cached data, so that the lower-level controller 20 that accesses other cached data can adjust the cache state of the accessed data.

[0032] In this application, the upper-level controller 30 uniformly undertakes the function of forwarding consistent messages across accelerators 10, so that the lower-level controller 20 only needs to send a consistent message to the upper-level controller 30 once. It does not need to know and contact each lower-level controller 20 in the cluster that may hold a copy of the accessed data, which effectively simplifies the operation of the lower-level controller 20 and reduces the communication overhead between accelerators 10 for maintaining consistency.

[0033] The final controller 20 may include a first remote agent module (RP) and a first local agent module (LP). The first remote agent module stores a remote directory for recording cache information of the remote accelerator on the cache of each acceleration submodule 101 in the local accelerator. The first local agent module stores a local directory for recording cache information of each acceleration submodule 101 in the local accelerator on the remote accelerator.

[0034] Considering that local address access and remote address access require different information queries and trigger different consistency actions, local address access mainly focuses on how to trigger updates to the local memory-bound directory and how to notify remote cache status updates, while remote address access needs to first determine whether a usable cache copy already exists locally. Therefore, in this application, the consistency management function within the final-level controller is split into a remote proxy module for remote addresses and a local proxy module for local addresses. Both modules use independent directory records, which avoids information mixing and reduces query efficiency.

[0035] The final controller 20 is used to send the first memory access request to the first local proxy module when the access address belongs to the local address; the first local proxy module is used to send the first memory access request to the directory bound to the local memory to access the local memory data in order to obtain the access data.

[0036] Figure 2 This is a schematic diagram illustrating a local memory access method provided in an embodiment of this application. Figure 2 The document lists three accelerators: accelerator a, accelerator b, and accelerator c. Each accelerator deploys two acceleration submodules and one final-level controller. Each acceleration submodule contains a cache. The final-level controllers in the first two accelerators are connected to the same upper-level controller, while the third accelerator is connected to the second upper-level controller. Figure 2 The example uses four upstream controllers and illustrates their connection relationship. Each downstream controller contains a first local agent module and a first remote agent module; each upstream controller contains a second local agent module and a second remote agent module. For ease of use... Figure 2 As shown in the diagram, the first local proxy module can be referred to as LocalProxy1, the first remote proxy module as RemoteProxy1, the second local proxy module as LocalProxy2, and the second remote proxy module as RemoteProxy2.

[0037] Figure 2 The memory access request issued by the first acceleration submodule is forwarded to the first local proxy module after being determined to be a local address by the final-level controller. The first local proxy module forwards the request to the directory bound to the local memory to complete the data reading or writing. At the same time, it sends consistency messages to other acceleration submodules that cache the data in parallel through the upper-level controller to synchronize their cache states. After the data reading or writing is completed, the access result is returned to the acceleration submodule that initiated the request along the original path through the first local proxy module, completing the entire process of one local memory access.

[0038] In this application, the sending of consistency messages is processed in parallel with the data access in local memory, instead of waiting for the memory access to be completed before sending the consistency message. This can effectively avoid increased access latency due to serial waiting.

[0039] The final controller 20 is used to send the access request to the first remote agent module when the access address belongs to a remote address.

[0040] The first remote proxy module is used to determine whether other caches inside the local accelerator record access data corresponding to the access address; if other caches inside the local accelerator record access data corresponding to the access address, it retrieves the access data from other caches; if other caches inside the local accelerator do not record access data corresponding to the access address, it sends a first memory access request to the upper-level controller 30 connected to it; and receives the access data fed back by the upper-level controller 30.

[0041] Figure 3 This application provides a schematic diagram of a remote access method where the local accelerator contains access data, and its architecture is similar to... Figure 2 The architectures are the same, differing only in data access paths; further details on the architecture will not be provided. For ease of description, we can... Figure 3 The leftmost acceleration submodule is considered the first acceleration submodule, and the acceleration submodule adjacent to it is considered the second acceleration submodule. Memory access requests from the first acceleration submodule are forwarded to the first remote proxy module after being identified as remote addresses by the final-level controller. The first remote proxy module checks whether other caches within the local accelerator record the access data corresponding to the access address. Assuming the second acceleration submodule within the local accelerator has the access data corresponding to the access address in its cache, the first remote proxy module retrieves the access data from its cache. The first remote proxy module then sends consistency messages in parallel to other acceleration submodules that have the data in their caches through the upper-level controller, synchronizing their cache states. After retrieving the access data, the first remote proxy module returns it to the requesting acceleration submodule along the original path.

[0042] Figure 4 This application provides a schematic diagram of remote access where the local accelerator does not have access data, and its architecture is similar to... Figure 2 The architectures are the same, differing only in data access paths; further details on the architecture will not be provided. For ease of description, we can... Figure 4The leftmost acceleration submodule is considered the first acceleration submodule. Memory access requests from this submodule are forwarded to the first remote proxy module after being identified as remote addresses by the final-level controller. The first remote proxy module checks if other caches within the local accelerator record the access data corresponding to the access address. If no other caches within the local accelerator record the access data, it sends a memory access request to its connected upper-level controller. The upper-level controller forwards this memory access request to the remote final-level controller that matches the access address. This remote final-level controller retrieves the access data from memory and feeds it back to the upper-level controller. The upper-level controller then returns the access data along the original path to the acceleration submodule that initiated the request. The first remote proxy module receives the access data from the upper-level controller and simultaneously sends consistency messages to other acceleration submodules that have the data in their caches, synchronizing their cache states.

[0043] Multiple accelerators 10 in the accelerator 10 cluster are allocated address ranges according to a unified address encoding method, for example, each accelerator in the cluster is allocated a continuous address number range.

[0044] The upper-level controller 30 determines the remote end-level controller 20 corresponding to the access address based on the address range to which the access address corresponding to the first memory access request belongs.

[0045] The upper-level controller 30 is used to forward the received first memory access request to the remote end-level controller 20, so that the remote end-level controller 20 can obtain access data from the cache or local memory of the acceleration submodule 101 connected to it, and feed back the access data to the upper-level controller 30; and forward the access data fed back by the remote end-level controller 20 to the first remote agent module of the end-level controller 20.

[0046] By adopting a unified address encoding and dividing the accelerators into ranges, the upper-level controller 30 can determine the target accelerator simply by matching the address ranges when forwarding memory access requests, without having to maintain a mapping table that corresponds to each specific address and accelerator. Even with a large number of accelerators, the address assignment can still be determined with low storage overhead and high efficiency.

[0047] In this application, if other caches within the accelerator already record the accessed data, it means that other acceleration submodules have previously retrieved and cached the same data from the remote end. In this case, the accessed data is retrieved directly from that cache without needing to initiate a cross-accelerator access through the upper-level controller again, effectively reducing the access path length and latency. If the accessed data is not cached within the accelerator, the first remote proxy module then initiates a memory access request to the upper-level controller. By first querying within the accelerator and then deciding whether to access across accelerators, existing data replicas in the cluster can be utilized preferentially, reducing the number of remote accesses.

[0048] The upper-level controller 30 includes a remote agent module and a local agent module. To distinguish them from the remote agent module and local agent module included in the lower-level controller 20, the remote agent module included in the upper-level controller 30 can be called the second remote agent module, and the local agent module included in the upper-level controller 30 can be called the second local agent module. The remote directory where the second remote agent module stores data is called the second remote directory; the local directory where the second remote agent module stores data is called the second local directory.

[0049] The second remote proxy module stores the second remote directory, which is used to record the cache information of the remote accelerator on the local accelerator; the second local proxy module stores the second local directory, which is used to record the cache information of the local accelerator on the remote accelerator.

[0050] The second remote agent module and the second local agent module set in the upper-level controller 30 are structurally similar to the first remote agent module and the first local agent module set in the lower-level controller. The difference is that the object managed by the lower-level controller 20 is the acceleration submodule 101, while the object managed by the upper-level controller 30 is the accelerator 10.

[0051] By using remote and local proxies with the same structure at different levels to manage cached information for objects of different granularities, the system's consistency management has good consistency and reusability, and it is also easy for the system to be further expanded to more levels.

[0052] The upper-level controller 30, upon receiving a first memory access request, determines whether other accelerators 10 connected to it have recorded access data corresponding to the access address. If the first target accelerator connected to it has recorded access data corresponding to the access address, the first memory access request is sent to the second local proxy module. If none of the other accelerators 10 connected to it have recorded access data corresponding to the access address, the first memory access request is sent to the second remote proxy module.

[0053] The second local agent module is used to send a first memory access request to the final controller 20 of the first target accelerator and to receive access data fed back by the final controller 20 of the first target accelerator.

[0054] The second remote proxy module is used to send a first memory access request to the second local proxy module of the target upper-level controller 30 whose access address matches. The second local proxy module of the target upper-level controller 30 is used to forward the first memory access request to the final controller 20 of the second target accelerator whose access address matches. The second remote proxy module receives the access data fed back by the final controller 20 of the second target accelerator.

[0055] In this application, the upper-level controller first checks whether there is available data within the scope of the accelerators it directly manages. Only when it is confirmed that there is no available data within its own management scope will the request continue to be routed to other upper-level controllers. This keeps access within a smaller management scope as much as possible, improves access efficiency, and reduces the number of accesses that cross a larger network.

[0056] As can be seen from the above technical solutions, by deploying a final-level controller within the accelerator and a higher-level controller between multiple accelerators, the cache consistency management of a large-scale accelerator cluster is split into two levels: one within the accelerator and the other between accelerators. Each level controller only needs to maintain limited information matching its management scope, avoiding the problem of centralized directories expanding rapidly as the cluster size increases. By setting proxy modules for local and remote addresses in the final-level and higher-level controllers respectively, and prioritizing the determination of whether available cached data exists within the current accelerator or its management scope during remote access, accesses that originally required cross-accelerator communication can be transformed into accesses within the accelerator or within the nearest range, reducing the number of cross-accelerator communications. Combined with unified address encoding and the method of dividing the accelerator by range, the controller does not need to maintain address mapping relationships when determining the access address, further reducing storage and query overhead. Overall, the solution provided in this application embodiment can complete data reading and state synchronization with low consistency maintenance overhead and short access paths even with a significant increase in the number of accelerators and acceleration submodules, thereby improving the overall data access efficiency and system scalability of a large-scale accelerator cluster.

[0057] Example 2: Taking a final-level controller as an example, when the data accessed by other acceleration submodules 101 in the cluster happens to belong to the accelerator 10 to which the final-level controller belongs, the accelerator to which the final-level controller belongs can be referred to as this accelerator. Memory access requests are forwarded by the upper-level controller 30 to the final-level controller 20 within this accelerator. For this final-level controller 20, the memory access request is a request forwarded from the outside. To facilitate the distinction between memory access requests initiated by acceleration submodules 101 directly connected to the final-level controller 20, in this application, memory access requests initiated by acceleration submodules 101 directly connected to the final-level controller 20 are referred to as first memory access requests, and requests forwarded from the outside are referred to as second memory access requests.

[0058] The above content is based on the example of an acceleration submodule 101 actively initiating a first memory access request. The following content will be based on the example of the final controller 20 connected to the acceleration submodule 101 receiving a second memory access request.

[0059] The final-level controller 20 receives a second memory access request sent by the upper-level controller 30 and determines the address type of the access address carried in the second memory access request based on the address mapping relationship. If the access address is a local address, the access data is retrieved from local memory. If the access address is a remote address, the access data is retrieved from the acceleration submodule 101 connected to it; the access data is then fed back to the upper-level controller 30. The second memory access request is a memory access request sent by other acceleration submodules 101 to the upper-level controller 30 through their connected final-level controller 20.

[0060] The final-level controller 20 also determines the address type based on the address mapping relationship when processing the second memory access request, and adopts the same local or remote data acquisition method as when processing the first memory access request. This is because, in the scheme of this application, each acceleration submodule 101 may be both a data requester and a data holder. When a remote acceleration submodule needs to access data belonging to other accelerators, it must rely on the final-level controller within those other accelerators to complete the data acquisition from local memory or local cache, and then feed the result back to the upper-level controller, which forwards it back to the side that initiated the request. By allowing the final-level controller 20 to use the same address judgment and data acquisition logic internally and externally, there is no need to design different processing flows for the two types of requests, simplifying the complexity of the data access process.

[0061] In this embodiment, the final-level controller has the ability to process second memory access requests forwarded from the upper-level controller. This allows data on any accelerator to be accessed not only by the local acceleration submodule but also by acceleration submodules of other accelerators in the cluster through a unified address determination process. This avoids the need to design additional processing logic due to different request sources, reducing the design and implementation complexity of the final-level controller. It enables the entire system to present the ability for data to be uniformly accessed by any acceleration submodule, further improving the flexibility of data sharing and overall access efficiency in large-scale accelerator clusters.

[0062] Example 3: The final controller 20 includes a protocol conversion module, which is used to convert the status information carried by the first memory access request according to the set status category; wherein, the set status category may include exclusive status (M), shared status (S) or invalid status (I).

[0063] Exclusive state means that the accessed data exists only in the current cache, and the same data does not exist in other caches; shared state means that the accessed data exists not only in the current cache, but also in other caches; invalid state means that the data in the cache can no longer be used.

[0064] Since the original caching protocols inside each acceleration submodule 101 may not be exactly the same, and the status identification methods they adopt may also differ, the protocol conversion module converts the status information from different sources into three standard states: exclusive, shared, and invalid. This allows the final controller 20 and the upper controller 30 to make judgments and processes based on these three unified state types, without having to adapt to each acceleration submodule 101's internal protocol separately, thus improving the system's compatibility with different caching protocols.

[0065] The final controller 20 is used to indicate that the acceleration submodule 101 that has obtained the access data may have specific requirements for the access data, such as modifying the data, when the status information is in the exclusive state. In order to prevent other acceleration submodules 101 from continuing to use the old data, it is necessary to notify other caches holding copies of the data to adjust their status to invalid. At this time, a consistency message that the access data has been adjusted to the invalid state can be sent to the upper controller 30 to which it is connected.

[0066] The final controller 20 is used to indicate that the data can still be read by multiple acceleration submodules at the same time when the status information is in the shared state. At this time, it can send a consistency message to the upper controller 30 connected to it to adjust the access data to the shared state. While ensuring data consistency, it tries to retain reusable cached data and reduce unnecessary data re-acquisition.

[0067] The final controller 20 is used to adjust the cache state of accessed data based on the consistency message sent by the upper controller 30.

[0068] In the specific implementation, the final controller 20 is used to forward the consistency message to the target acceleration submodule that caches the access data when the consistency message is invalid; the target acceleration submodule is used to delete the access data based on the consistency message.

[0069] By forwarding the consistency message of the invalid state to the target acceleration submodule that actually holds the expired data, and having the target acceleration submodule actively delete the data instead of having the last-level controller clear it uniformly, it is ensured that the state adjustment occurs in the acceleration submodule that actually stores the data. This effectively avoids the situation where the last-level controller thinks the data is invalid, but the acceleration submodule's local cache still retains the old data.

[0070] In this embodiment, a protocol conversion module is introduced to uniformly convert various caching protocols that may differ within the acceleration submodules into three standard states: exclusive, shared, and invalid. This allows the final-level controller and the upper-level controller to process state information from different acceleration submodules in a unified manner, improving the system's compatibility with multiple caching protocols. By distinguishing between exclusive and shared states and triggering different consistency messages, other replicas are promptly invalidated when data may be modified, while other replicas are retained when data is still shareable and readable, ensuring data consistency and the reusability of cached data. By forwarding invalid state consistency messages level by level to the target acceleration submodule that actually holds the data and performs the deletion operation, the accuracy of cache state adjustments is ensured, avoiding the risk of data inconsistency. While ensuring data consistency in a large-scale accelerator cluster, unnecessary data invalidation and duplicate retrieval are minimized, improving the utilization efficiency of cache resources.

[0071] Example 4: Depending on deployment requirements, the upper-level controller 3 may include multiple levels of controllers. These multiple levels of controllers may include multiple first-level controllers and at least one second-level controller; each first-level controller is used to connect to multiple different final-level controllers 20; the second-level controller is used to connect to multiple first-level controllers. The final-level controller, i.e., the final-level controller 20, is deployed inside each accelerator 10 and is used to manage the consistency between multiple acceleration submodules 101 within that accelerator 10. First-level controllers connect to several final-level controllers 20, and second-level controllers then connect to several first-level controllers, and so on, expanding upwards level by level.

[0072] This application is not limited to a two- or three-tier architecture and can be extended to more tiers. When the number of accelerators 10 in the cluster reaches hundreds, thousands, or even tens of thousands, if a single-level upper-level controller 30 directly connects to all the lower-level controllers 20, the upper-level controller 30 needs to maintain cache information related to all accelerators 10 at the same time, and its storage and query pressure will increase with the cluster size.

[0073] This application introduces multiple layers, with each level controller only needing to connect to and manage a relatively limited number of lower-level controllers. This distributes the management pressure, which was originally concentrated on a single node, to multiple layers and multiple nodes, enabling the system to continuously support larger-scale accelerator cluster expansion by increasing the number of layers.

[0074] Figure 5 This is a schematic diagram of a structure for interconnecting multiple GPUs, provided in an embodiment of this application. Taking 64 accelerators as an example, each accelerator contains 4 acceleration sub-modules and is managed by a final-level controller. Each upper-level controller can connect to the 4 final-level controllers, and the 4 final-level controllers and the upper-level controllers can be interconnected through a mesh topology, thereby completing the interconnection of the 64 accelerator cluster.

[0075] The multiple acceleration sub-modules 101 inside the accelerator 10 are connected by a fully interconnected topology; the multiple final-level controllers 20 are connected to the connected upper-level controllers 30 by a tree structure; and the uppermost upper-level controllers 30 among the multiple upper-level controllers 30 are connected by a mesh topology.

[0076] Figure 6 This application provides a schematic diagram of a hierarchical topology interconnection of a data access system. Figure 6 It demonstrates a hybrid topology consisting of full interconnection between acceleration submodules, tree-like connections between the final-level controller and the upper-level controller, and mesh connections between the top-level upper-level controllers.

[0077] In this embodiment, the number of acceleration submodules within the same accelerator is relatively limited. A fully interconnected approach allows direct communication between any two acceleration submodules via the shortest path, minimizing data access latency within the accelerator. A tree structure supports a large number of end-level controllers with fewer connections. The tree structure between end-level and upper-level controllers facilitates hierarchical organization and cluster expansion. As the hierarchy converges upwards, the top-level nodes have the most concentrated interconnection needs. The mesh topology ensures efficient interconnection between a sufficient number of top-level controllers without being limited by the outgoing bandwidth of a single node, preventing the top layer from becoming a bottleneck in the entire interconnection network. By employing topologies that match the characteristics of different levels and sizes of connected objects, a good balance is achieved between access latency and scalability in the interconnection network of the entire large-scale accelerator cluster.

[0078] Figure 7 A flowchart illustrating a data access method for a final-level controller provided in this application embodiment is provided. This method is applicable to a final-level controller deployed within an accelerator and connected to multiple acceleration submodules within the accelerator. The method includes: S701: Receives the first memory access request sent by the acceleration submodule and determines the address type of the access address carried in the first memory access request based on the address mapping relationship.

[0079] The final-level controller here includes a first remote proxy module and a first local proxy module. The first remote proxy module stores a remote directory to record cache information of the remote accelerator on the caches of each acceleration submodule within the local accelerator. The first local proxy module stores a local directory to record cache information of each acceleration submodule within the local accelerator on the remote accelerator. The specific functions of both are consistent with the description in the previous embodiments. The address type can include local addresses and remote addresses.

[0080] S702: When the access address is a local address, retrieve the access data from local memory.

[0081] The final controller sends the first memory access request to its internally deployed first local agent module. The first local agent module then forwards the memory access request to the directory bound to the local memory to access the local memory and retrieve the data. This step completes the data retrieval directly locally, resulting in the shortest access path and the lowest latency for accessing local addresses.

[0082] S703: If the access address is a remote address, obtain access data from other acceleration submodules that match the access address.

[0083] If the access address is a remote address, the final controller will send the first memory access request to its internally deployed first remote proxy module. The first remote proxy module then determines whether other caches within the local accelerator have recorded the access data corresponding to the access address. If other caches within the local accelerator have recorded the access data corresponding to the access address, it retrieves the access data from those caches (corresponding to...). Figure 3 (This illustrates the scenario of remote access with locally existing access data). If other caches within the local accelerator do not record the access data corresponding to the access address, a first memory access request is sent to its connected upper-level controller. The upper-level controller forwards the received first memory access request to the remote final-level controller, allowing the remote final-level controller to retrieve the access data from the cache of its connected acceleration submodule or its local memory, and then feeds the access data back to the upper-level controller. The upper-level controller then forwards the access data fed back by the remote final-level controller to the first remote proxy module of the final-level controller (corresponding to...). Figure 4 (This refers to the scenario where remote access occurs but the accessed data is not available locally). This step prioritizes determining if there is an available cache within the accelerator, converting remote access into intra-accelerator access as much as possible, thus reducing the frequency of cross-accelerator communication.

[0084] S704: Feeds back the access data to the acceleration submodule and sends a consistency message to the upper controller it is connected to, so that the upper controller can forward the consistency message to the last controllers that cache the access data, so that the last controllers that cache the access data can adjust the cache state of the access data.

[0085] Multiple accelerators in an accelerator cluster are allocated address ranges according to a unified address encoding method.

[0086] The upper-level controller can determine the remote end-level controller corresponding to the access address based on the address range to which the access address belongs to the first memory access request, thereby completing the location of the remote end-level controller with lower storage and query overhead.

[0087] In some embodiments, it also includes: Receive a second memory access request sent by the upper-level controller, and determine the address type of the access address carried in the second memory access request based on the address mapping relationship; wherein, the second memory access request is a memory access request sent by other acceleration submodules to the upper-level controller through the final controller they are connected to. If the access address is a local address, the accessed data is retrieved from local memory; If the access address is a remote address, the access data is obtained from the acceleration submodule connected to it; the access data is then fed back to the upper-level controller.

[0088] In some embodiments, after receiving the first memory access request sent by the acceleration submodule, the method further includes: The status information carried by the first memory access request is transformed according to the set status category; the set status categories include exclusive status, shared status, or invalid status.

[0089] In some embodiments, sending a consistency message to the upstream controller to which it is connected includes: When the status information is in an exclusive state, a consistency message is sent to the upstream controller to which it is connected, adjusting the access data to an invalid state. When the status information is in a shared state, a consistency message is sent to the upstream controller to which it is connected to adjust the access data to a shared state.

[0090] In some embodiments, it also includes: Upon receiving a consistency message from the superior controller, the cache state of the accessed data is adjusted according to the consistency message.

[0091] In some embodiments, adjusting the cache state of accessed data based on consistency messages includes: If the consistency message is invalid, the consistency message is forwarded to the target acceleration submodule that caches the accessed data, so that the target acceleration submodule can delete the accessed data based on the consistency message.

[0092] The above processing method is consistent with the corresponding processing method of the final-level controller in the previous embodiment. By distinguishing between exclusive and shared states and triggering different consistency actions respectively, it takes into account both data consistency maintenance and the reusability of cached data.

[0093] For a description of the features in the embodiments corresponding to the data access method, please refer to the relevant descriptions in the embodiments corresponding to the data access system, which will not be repeated here.

[0094] As can be seen from the above technical solution, using a final-level controller to manage the internal cache consistency of the accelerator and a higher-level controller to manage the cache consistency between accelerators, each final-level controller only needs to maintain information on a limited number of acceleration sub-modules within its own accelerator, and the higher-level controller only needs to manage a limited number of final-level controllers connected to it. Compared to the method of a single control node centrally managing all acceleration sub-modules, this avoids the problem of the management information volume expanding rapidly with the total number of accelerators and acceleration sub-modules, thus giving the system better scalability and enabling it to support larger-scale accelerator clusters. Acceleration sub-modules only need to send access requests to the final-level controller of their respective accelerators, without needing to determine the data storage location themselves, simplifying the design complexity of the acceleration sub-modules. The final-level controller directly determines the local or remote type of the access address based on the address mapping relationship, without needing to perform global traversal searches one by one, shortening the latency required for address determination. When the access address is a local address, data is directly retrieved from local memory, resulting in the shortest access path and lowest latency. When accessing a remote address, data is preferentially retrieved from other acceleration submodules within the same accelerator that match that address, rather than directly accessing remote memory across accelerators. This fully utilizes existing data replicas, reduces unnecessary cross-accelerator memory accesses, and lowers communication pressure between accelerators. When the last-level controller reports accessed data, it only needs to send a consistency message once to its connected upper-level controller. The upper-level controller then forwards this message to other last-level controllers that cache the data, prompting them to adjust their cache states. This eliminates the need for the last-level controllers to notify each node in the cluster that may hold a copy of the data individually. This reduces the communication overhead required for consistency maintenance and ensures that the data used by each acceleration submodule in the cluster is always in a consistent state, avoiding the use of expired data replicas. While ensuring cache consistency in a large-scale accelerator cluster, this effectively reduces the storage and communication overhead of consistency management, shortens data access paths, and thus improves data access efficiency.

[0095] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] The data access system and method provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data access system, characterized in that, It includes multiple accelerators, a final-level controller deployed within the multiple accelerators, and at least one upper-level controller; wherein the upper-level controller is used to connect to the multiple final-level controllers; each accelerator includes multiple acceleration submodules; each acceleration submodule has its corresponding cache and local memory; The acceleration submodule is connected to the final-level controller deployed inside its accelerator, and is used to send a first memory access request to the final-level controller and receive access data fed back by the final-level controller; The final-level controller is configured to receive a first memory access request sent by the acceleration submodule, determine the address type of the access address carried in the first memory access request based on the address mapping relationship; wherein, the address type includes local address and remote address; if the access address is a local address, obtain access data from local memory; if the access address is a remote address, obtain the access data from other acceleration submodules that match the access address; feed the access data back to the acceleration submodule, and send a consistency message to the upper-level controller connected to it; The upper-level controller is used to forward the consistency message to other lower-level controllers that cache the access data, so that the other lower-level controllers that cache the access data can adjust the cache state of the access data.

2. The data access system according to claim 1, characterized in that, The final-level controller includes a first remote agent module and a first local agent module. The first remote agent module stores a remote directory to record cache information of the remote accelerator on the cache of each acceleration submodule in the local accelerator. The first local agent module stores a local directory to record cache information of each acceleration submodule in the local accelerator on the remote accelerator.

3. The data access system according to claim 2, characterized in that, The final-level controller is configured to send the first memory access request to the first local proxy module when the access address is a local address. The first local proxy module is used to send the first memory access request to the directory bound to the local memory to access local memory data in order to obtain the access data.

4. The data access system according to claim 2, characterized in that, The final controller is configured to send the first memory access request to the first remote proxy module when the access address belongs to a remote address. The first remote proxy module is used to determine whether other caches inside the local accelerator record the access data corresponding to the access address; if other caches inside the local accelerator record the access data corresponding to the access address, the module retrieves the access data from the other caches; if other caches inside the local accelerator do not record the access data corresponding to the access address, the module sends the first memory access request to the upstream controller connected to it. Receive access data fed back from the superior controller; The upper-level controller is configured to forward the received first memory access request to the remote end-level controller, so that the remote end-level controller can obtain access data from the cache or local memory of the acceleration submodule connected to it and feed the access data back to the upper-level controller; and forward the access data fed back by the remote end-level controller to the first remote proxy module of the end-level controller.

5. The data access system according to claim 4, characterized in that, Multiple accelerators in an accelerator cluster are allocated address ranges according to a unified address encoding method; The upper-level controller is used to determine the remote end-level controller corresponding to the access address based on the address range to which the access address corresponding to the first memory access request belongs.

6. The data access system according to claim 4, characterized in that, The upper-level controller includes a second remote agent module and a second local agent module; the second remote agent module stores a second remote directory for recording the cached information of the remote accelerator on the local accelerator; the second local agent module stores a second local directory for recording the cached information of the local accelerator on the remote accelerator.

7. The data access system according to claim 6, characterized in that, The upper-level controller is configured to, upon receiving the first memory access request, determine whether other accelerators connected to it have recorded the access data corresponding to the access address; if the first target accelerator connected to it has recorded the access data corresponding to the access address, send the first memory access request to the second local proxy module; if none of the other accelerators connected to it have recorded the access data corresponding to the access address, send the first memory access request to the second remote proxy module. The second local proxy module is used to send the first memory access request to the final controller of the first target accelerator; and to receive access data fed back by the final controller of the first target accelerator. The second remote proxy module is used to send the first memory access request to the second local proxy module of the target upper-level controller whose access address matches; The second local agent module of the target upper-level controller is used to forward the first memory access request to the final controller of the second target accelerator whose access address matches; and to receive access data fed back by the final controller of the second target accelerator.

8. The data access system according to claim 1, characterized in that, The final-level controller is configured to receive a second memory access request sent by the upper-level controller, determine the address type of the access address carried in the second memory access request based on the address mapping relationship, obtain access data from local memory if the access address is a local address, obtain the access data from the acceleration submodule connected to it if the access address is a remote address, and feed the access data back to the upper-level controller; wherein, the second memory access request is a memory access request sent by other acceleration submodules to the upper-level controller through their connected final-level controller.

9. The data access system according to claim 1, characterized in that, The final controller includes a protocol conversion module, which is used to convert the status information carried by the first memory access request according to a set status category; wherein the set status category includes exclusive status, shared status, or invalid status.

10. The data access system according to claim 9, characterized in that, The final-level controller is configured to send a consistency message indicating that the access data has been adjusted to an invalid state to its connected upper-level controller when the status information is in an exclusive state; and to send a consistency message indicating that the access data has been adjusted to a shared state to its connected upper-level controller when the status information is in a shared state.

11. The data access system according to claim 10, characterized in that, The last-level controller is used to adjust the cache state of the accessed data based on the consistency message sent by the upper-level controller.

12. The data access system according to claim 11, characterized in that, The final controller is configured to forward the consistency message to the target acceleration submodule that caches the access data when the consistency message is invalid. The target acceleration submodule is used to delete the access data based on the consistency message.

13. The data access system according to any one of claims 1-12, characterized in that, The upper-level controller includes multiple levels of controllers; the multiple levels of controllers include multiple first-level controllers and at least one second-level controller; each first-level controller is used to connect to multiple different final-level controllers; the second-level controller is used to connect to multiple first-level controllers.

14. The data access system according to claim 13, characterized in that, The multiple acceleration sub-modules within the accelerator are connected by a fully interconnected topology; the multiple final-level controllers are connected to their connected upper-level controllers by a tree structure; and the uppermost upper-level controller among the multiple upper-level controllers is connected by a mesh topology.

15. A data access method, characterized in that, Applicable to a final-stage controller, the final-stage controller being deployed within an accelerator and connected to multiple acceleration submodules within the accelerator; the method includes: The system receives a first memory access request sent by the acceleration submodule and determines the address type of the access address carried in the first memory access request based on the address mapping relationship; wherein, the address type includes local address and remote address; If the access address is a local address, the access data is retrieved from local memory; If the access address is a remote address, the access data is obtained from other acceleration submodules that match the access address; The access data is fed back to the acceleration submodule and a consistency message is sent to the upper-level controller connected to it, so that the upper-level controller forwards the consistency message to other end-level controllers that cache the access data, so that the other end-level controllers that cache the access data can adjust the caching state of the access data.

Images