A multi-element heterogeneous distributed system, memory access method and storage medium

Abstract

The application relates to the technical field of data storage, and discloses a multi-element heterogeneous distributed system, a memory access method and a storage medium. A first uniform interconnection bus unit receives a read request sent by a first device through a protocol adaptation interface module, sets a cache state of a first memory unit as an invalid state, and sends a request message obtained by encapsulating the read request to a second uniform interconnection bus unit of a second device. The second uniform interconnection bus unit reads data from a second memory unit, sets a cache state of the second memory unit as a shared state, and sends a response message to the first uniform interconnection bus unit. The first uniform interconnection bus unit stores data carried in the response message to the first memory unit, and sets the cache state of the first memory unit as the shared state. By deploying the uniform interconnection bus unit on each device, consistent memory communication between the devices is realized, memory sharing is realized, and the performance of the multi-element heterogeneous distributed system is improved.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a multi-element heterogeneous distributed system, a memory access method, and a storage medium. Background Technology

[0002] With the slowdown of Moore's Law, large-scale parallel processing of massive amounts of data can no longer rely on a single device for rapid completion. Instead, it requires combining many different types of devices for processing—a process known as heterogeneous processing. To ensure effective communication among these devices, various cache coherence interconnect protocol standards have been proposed.

[0003] Different types of processors use different off-chip cache coherency buses for interconnecting with input / output (I / O) peripherals. Furthermore, the interconnect buses between processors are only for processors of the same type and cannot be used for interconnecting other processors. Because different types of processors use different interconnect buses, they cannot communicate with each other. Processors and I / O peripherals also use different bus protocols, requiring different processing. This current interconnect topology requires implementing multiple types of interconnect buses, resulting in a complex implementation structure. Moreover, memory cannot be shared between different types of devices, memory expansion access is limited, and the optimal performance of a diverse, heterogeneous distributed system cannot be achieved.

[0004] It is evident that improving the performance of diverse heterogeneous distributed systems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-element heterogeneous distributed system, a memory access method, and a storage medium, which can solve the problem of low performance in multi-element heterogeneous distributed systems.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a multi-element heterogeneous distributed system, comprising at least two processors and at least one accelerator; wherein each processor and accelerator has its own corresponding memory unit; each processor and accelerator is equipped with a unified interconnect bus unit; a protocol adaptation interface module is deployed on the unified interconnect bus unit; the protocol adaptation interface module is used to implement the conversion between the consistent protocol interface of different types of devices and the unified interconnect bus protocol interface; The first unified interconnect bus unit is configured to receive a read request sent by the first device through the protocol adaptation interface module, set the cache state of the first memory unit of the first device to an invalid state, encapsulate the read request into a request message conforming to a set message format and send it to the second unified interconnect bus unit of the second device; wherein the first device and the second device are any two devices among the processors and the accelerators; The second unified interconnect bus unit is configured to, upon receiving the request message, read data from the second memory unit of the second device; set the cache state of the second memory unit to a shared state; encapsulate the data into a response message conforming to the message format; and send the response message to the first unified interconnect bus unit. The first unified interconnect bus unit is used to store the data carried in the received response message into the first memory unit; and to set the cache state of the first memory unit to a shared state according to the cache state carried in the response message.

[0007] On one hand, the unified interconnect bus unit includes a protocol layer, an adaptation layer, and a physical layer; wherein, the protocol layer includes a protocol adaptation interface module, a request queue management module, a response queue management module, a message parsing and encapsulation module, and a clock domain conversion module; wherein, the protocol adaptation interface module is connected to the request queue management module and the response queue management module respectively, and is used to transmit the received request to the request queue management module or the response queue management module according to the request type; the message parsing and encapsulation module is connected to the request queue management module, the response queue management module, and the clock domain conversion module respectively, and is used to implement message encapsulation and decapsulation.

[0008] On the one hand, the first protocol adaptation interface module of the first unified interconnect bus unit is used to receive the read request sent by the first device and transmit the read request to the first request queue management module through the internal bus; The first request queue management module is configured to receive the read request; set the cache state of the first memory unit of the first device to an invalid state; query the address of the second device matching the second identifier from the node forwarding table according to the second identifier of the second device carried in the read request; and transmit the read request and the address of the second device to the first packet parsing and encapsulation module. The first message parsing and encapsulation module is used to encapsulate the received read request according to the message format to obtain the request message; and to transmit the request message and the second device address to the first clock domain conversion module. The first clock domain conversion module is used to transmit the request message and the second device address to the hub through the first adaptation layer and the first physical layer; The hub is used to forward the request message to the second device corresponding to the second device address.

[0009] On the one hand, the first protocol adaptation interface module of the first unified interconnect bus unit is used to receive the read request sent by the first device and transmit the read request to the first request queue management module through the internal bus; The first request queue management module is used to receive the read request; set the cache state of the first memory unit of the first device to an invalid state; and transmit the read request to the first message parsing and encapsulation module. The first message parsing and encapsulation module is used to encapsulate the received read request according to the message format to obtain the request message; and to transmit the request message to the first clock domain conversion module. The first clock domain conversion module is used to transmit the request message to the hub through the first adaptation layer and the first physical layer; The hub is configured to query the address of the second device that matches the second identifier from the node forwarding table based on the second identifier of the second device carried in the request message; and forward the request message to the second device corresponding to the second device address.

[0010] On one hand, the first message parsing and encapsulation module is used to encapsulate the first identifier of the first device, the second identifier of the second device to which the read request points, the message type and message sequence number to which the read request belongs, and the second memory address according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal, so as to obtain the request message; wherein, the second memory address carries a status indicator that the first memory unit is in an invalid state.

[0011] On one hand, the second clock domain conversion module of the second unified interconnect bus unit is used to receive the request message transmitted by the hub through the second physical layer and the second adaptation layer, and forward the request message to the second message parsing and encapsulation module; The second message parsing and encapsulation module is used to parse the request message to obtain the first identifier of the first device, the message sequence number, and the second memory address; and send the second memory address to the second response queue management module. The second response queue management module is used to read data from the second memory unit according to the second memory address, and transmit the data to the second message parsing and encapsulation module; and set the cache state of the second memory unit to a shared state; The second message parsing and encapsulation module is used to receive the data; query the node forwarding table for the first device address that matches the first identifier according to the first identifier; encapsulate the second identifier, the first identifier, the length of the data, the message type to which the response request belongs, the message sequence number, the first memory address, the data, and the write enable signal according to the message format to obtain the response message; and transmit the response message and the first device address to the second clock domain conversion module; wherein, the first memory address carries a status identifier that the second memory unit is in a shared state; The second clock domain conversion module is used to transmit the response message and the first device address to the hub through the second adaptation layer and the second physical layer; The hub is used to forward the response message to the first device corresponding to the first device address.

[0012] On one hand, the second clock domain conversion module of the second unified interconnect bus unit is used to receive the request message transmitted by the hub through the second physical layer and the second adaptation layer, and forward the request message to the second message parsing and encapsulation module; The second message parsing and encapsulation module is used to parse the request message to obtain the first identifier of the first device, the message sequence number, and the second memory address; and send the second memory address to the second response queue management module. The second response queue management module is used to read data from the second memory unit according to the second memory address and transmit the data to the second message parsing and encapsulation module; the cache state of the second memory unit is set to a shared state; The second message parsing and encapsulation module is used to receive the data; query the first memory address matched by the first identifier from the node forwarding table according to the first identifier; encapsulate the second identifier, the first identifier, the length of the data, the message type to which the response request belongs, the message sequence number, the first memory address, the data, and the write enable signal according to the message format to obtain the response message; and transmit the response message to the second clock domain conversion module; wherein, the first memory address carries a status identifier indicating that the second memory unit is in a shared state; The second clock domain conversion module is used to transmit the response message to the hub through the second adaptation layer and the second physical layer; The hub is configured to query the first device address matching the first identifier from the node forwarding table based on the first identifier carried in the response message; and forward the response message to the first device corresponding to the first device address.

[0013] On one hand, the first unified interconnect bus unit is used to encapsulate the first identifier, the second identifier, the message type to which the monitoring belongs, and the message sequence number according to the message format to obtain a monitoring message; and to transmit the monitoring message and the second memory address to the first clock domain conversion module. The first clock domain conversion module is used to transmit the listening message and the second device address to the hub through the first adaptation layer and the first physical layer; The hub is used to forward the listening message to the second device corresponding to the second device address.

[0014] On one hand, the first unified interconnect bus unit is used to encapsulate the first identifier, the second identifier, the message type to which the monitoring belongs, and the message sequence number according to the message format to obtain a monitoring message; and transmit the monitoring message to the first clock domain conversion module; The first clock domain conversion module is used to transmit the monitoring message to the hub through the first adaptation layer and the first physical layer; The hub is configured to query the address of the second device that matches the second identifier from the node forwarding table based on the second identifier of the second device carried in the monitoring message; and forward the monitoring message to the second device corresponding to the second device address.

[0015] On one hand, the third unified interconnect bus unit is used to receive a write request sent by the third device, set the cache state of the third memory unit of the third device to an invalid state, encapsulate the data to be written into a write message conforming to the message format, and send the write message to the first unified interconnect bus unit through the hub; The first unified interconnect bus unit is configured to receive the write message fed back by the hub, store the data to be written carried in the write message into the first memory unit; set the cache state of the first memory unit to a unique clean state according to the cache state carried in the write message; and send a monitoring message to the second unified interconnect bus unit through the hub. The second unified interconnect bus unit is used to receive the monitoring message sent by the hub; and to set the cache state of the second memory unit to invalid state according to the invalid state carried in the monitoring message.

[0016] On the one hand, the third protocol adaptation interface module of the third unified interconnect bus unit is used to receive the write request sent by the third device and transmit the write request to the third request queue management module through the internal bus; The third request queue management module is used to receive the write request; set the cache state of the third memory unit of the third device to an invalid state; query the first device address matching the first identifier from the node forwarding table according to the first identifier of the first device carried in the write request; and transmit the write request and the first device address to the third packet parsing and encapsulation module. The third message parsing and encapsulation module is used to encapsulate the received write request according to the message format to obtain the write message; and to transmit the write message and the first device address to the third clock domain conversion module. The third clock domain conversion module is used to transmit the write message and the first device address to the hub through the third adaptation layer and the third physical layer; The hub is used to forward the write message to the first device corresponding to the first device address.

[0017] On the one hand, the third protocol adaptation interface module of the third unified interconnect bus unit is used to receive the write request sent by the third device and transmit the write request to the third request queue management module through the internal bus; The third request queue management module is used to receive the write request; set the cache state of the third memory unit of the third device to an invalid state; query the first device address matching the first identifier from the node forwarding table according to the first identifier of the first device carried in the write request; and transmit the write request and the first device address to the third packet parsing and encapsulation module. The third message parsing and encapsulation module is used to encapsulate the received write request according to the message format to obtain the write message; and to transmit the write message to the third clock domain conversion module. The third clock domain conversion module is used to transmit the write message to the hub through the third adaptation layer and the third physical layer; The hub is configured to query the address of the first device that matches the first identifier from the node forwarding table based on the first identifier of the first device carried in the write message; and forward the write message to the first device corresponding to the first device address.

[0018] On one hand, the third message parsing and encapsulation module is used to encapsulate the third identifier of the third device, the first identifier of the first device to which the write request points, the message type and message sequence number to which the write request belongs, the first memory address, and the data to be written according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal, so as to obtain the write message; wherein, the first memory address carries a status identifier that the third memory unit is in an invalid state.

[0019] On the one hand, both the adapter layer and the physical layer are provided with bypass units; wherein, the bypass units are used to realize direct connection between the adapter layer and the physical layer; When the first device and the second device are located on the same printed circuit board, the bypass units of the first adapter layer and the first physical layer of the first device and the second adapter layer and the second physical layer of the second device are in the open state to realize the direct connection between the first device and the second device.

[0020] On the one hand, it also includes a storage medium that is independent of each of the processors and the accelerators; the unified interconnect bus unit is deployed on the storage medium.

[0021] On the one hand, each of the unified interconnect bus units is used to determine the matching cache state based on the state of the data on its corresponding memory unit; wherein, the cache state includes invalid state, unique state and shared state; the unique state includes unique clean state, unique dirty state, unique clean free state and unique partially dirty state; the shared state includes shared clean state and shared dirty state.

[0022] This invention also provides a memory access method, including: The protocol adaptation interface module receives a read request from the first device and sets the cache state of the first memory unit of the first device to an invalid state; wherein, the protocol adaptation interface module is used to implement the conversion between the consistency protocol interface of different types of devices and the first unified interconnect bus protocol interface; Based on the first identifier of the first device and the second identifier of the second device to which the read request points, the read request is encapsulated into a first request message conforming to a set message format; The first request message is sent to the second unified interconnect bus unit of the second device; Receive the response message from the second unified interconnect bus unit and store the data carried in the response message in the first memory unit; Based on the cache status carried in the response message, the cache status of the first memory unit is set to a shared state.

[0023] On the one hand, it also includes: Receive a second request message sent by the hub; wherein the second request message is transmitted to the hub by the second unified interconnect bus unit of the second device; Data is read from the first memory unit according to the first memory address carried in the second request message; Set the cache state of the first memory unit to a shared state, and encapsulate the data into a response message conforming to the message format; The response message is sent to the second device via the hub.

[0024] On the one hand, it also includes: The device receives a write message from the hub and stores the data to be written carried in the write message into the first memory unit; wherein the write message is transmitted to the hub by the third unified interconnect bus unit of the third device. Based on the cache state carried in the write message, the cache state of the first memory unit is set to a unique clean state; The hub sends a monitoring message to the second unified interconnect bus unit so that the second unified interconnect bus unit can receive the monitoring message sent by the hub; and sets the cache state of the second memory unit to invalid state according to the invalid state carried in the monitoring message.

[0025] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the memory access method described above.

[0026] As can be seen from the above technical solution, the multi-heterogeneous distributed system includes at least two processors and at least one accelerator; each processor and accelerator has its own corresponding memory unit; each processor and accelerator is equipped with a unified interconnect bus unit; a protocol adaptation interface module is deployed on the unified interconnect bus unit; the protocol adaptation interface module is used to convert the consistency protocol interface of different types of devices to the unified interconnect bus protocol interface. The first device and the second device are any two devices among the processors and accelerators. The first unified interconnect bus unit deployed on the first device is used to receive read requests sent by the first device through the protocol adaptation interface module. Since there is currently no data in the first memory unit, the cache state of the first memory unit of the first device can be set to an invalid state first; the read request is encapsulated into a request message conforming to the set message format and sent to the second unified interconnect bus unit of the second device. The second unified interconnect bus unit is used to read data from the second memory unit of the second device after receiving the request message; since data sharing is required, the cache state of the second memory unit can be set to a shared state, and the data is encapsulated into a response message conforming to the message format; the response message is sent to the first unified interconnect bus unit. The first unified interconnect bus unit is used to store the data carried in the received response message into the first memory unit; and to set the cache state of the first memory unit to a shared state according to the cache state carried in the response message. The beneficial effect of this invention is that by deploying a unified interconnect bus unit on each device and deploying a protocol adapter interface module on the unified interconnect bus unit, compatibility with various types of devices can be achieved. This allows processors with different instruction sets, different types of accelerators, and different memory units to be unified into a single system, forming a supercomputer system. This enables memory sharing between different processors and between different processors and accelerators, forming a large memory pool and alleviating the memory wall and I / O wall problems. The unified interconnect bus unit can be used to interconnect different types of devices through different topologies, supporting flexible interconnect topologies and allowing for flexible scaling without affecting existing deployments. By recording the cache state of the memory unit, consistent memory communication between devices within the system is achieved. The deployment method of this invention greatly improves the performance of a multi-dimensional heterogeneous distributed system. Attached Figure Description

[0027] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the structure of a multi-element heterogeneous distributed system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a protocol adaptation interface module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an architecture for achieving cache coherency interconnection between multiple types of processors and multiple types of accelerators, provided as an embodiment of the present invention. Figure 4 A block diagram of internal functional modules of a unified interconnect bus unit provided in an embodiment of the present invention; Figure 5 A flowchart of a memory access method provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0030] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0031] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] With the advent of the artificial intelligence era, the memory wall and input / output (I / O) wall problems are becoming increasingly serious. The memory wall problem mainly stems from the fact that the growth rate of memory bandwidth cannot keep up with the rapidly increasing number of Central Processing Unit (CPU) cores, causing memory bandwidth to become a bottleneck. The I / O wall problem is mainly caused by insufficient memory capacity, resulting in a decline in the speed of accessing external storage. How to effectively expand memory to solve the memory wall and I / O wall problems has become an urgent issue for the industry.

[0033] The most common way to expand memory is by adding memory channels to the processor. Currently, 8 channels and even 12 channels are supported, but it's impossible to increase indefinitely. Each additional memory channel requires adding many more signals, which poses a significant challenge to processor power consumption, heat dissipation, packaging, and printed circuit board (PCB) design. Of course, some new storage media have emerged on the market, but the current Double Data Rate (DDR) interface is not compatible with a wide range of storage media.

[0034] The high-speed serial computer expansion bus standard (PCI-Express, PCIE) interface is currently the bus type commonly used by high-performance I / O devices. The PCIE 6.0 standard has been officially released, with speeds up to 128GBps (switching bandwidth). However, the PCIE interface does not support cache coherence transactions and also has the problem of memory address space isolation, making it impossible to expand host memory.

[0035] Furthermore, with the slowdown of Moore's Law, processing large amounts of data requires massive parallel processing, which can no longer be accomplished by relying on a single fast chip. It necessitates combining many types of processors, i.e., heterogeneous processing. How to enable effective communication between these devices also requires a new bus protocol.

[0036] Currently, commonly used off-chip cache coherence buses for interconnecting processors and I / O peripherals mainly include multi-protocol interconnect technology bus (Compute Express Link, CXL), cache coherence interconnect protocol for microprocessors (Advanced RISC Machines, ARM) for accelerators (CCIX), and bus and its communication protocol (NVlink).

[0037] Inter-processor interconnect buses mainly include the Ultra Path Interconnect (UPI) bus, the Infinity Fabric (IF) interconnect bus between AMD processors, ARM's Corelink series of on-chip buses, and custom coherence buses developed by domestic CPU manufacturers. Inter-processor interconnect buses enable access to memory between different processors and also facilitate memory expansion.

[0038] Currently, the bus interconnect protocols between processors and the interconnect protocols between processors and I / O peripherals typically use different protocols. Taking three types of processors and three types of accelerators as examples (for ease of description, the three types of processors can be referred to as Type I processors, Type II processors, and Type III processors), and the three types of accelerators as Field Programmable Gate Arrays (FPGAs), Graphics Processing Units (GPUs), and Digital Shoreline Analysis Systems (DSAS), respectively, as examples, Type I processors can use an Ultra Path Interconnect (UPI) bus to connect with each other; Type II processors can use an on-chip bus (Corelink); and Type III processors can use a Coherence Bus (HCCS). Type I processors and FPGAs use a Multi-Protocol Interconnect (CXL) bus; Type II processors and GPUs use a Cache Coherence Interconnect (CCIX) bus; and Type III processors and DSASs use a High-Speed ​​Serial Computer Expansion Bus (PCIE).

[0039] For off-chip cache coherence interconnect protocols, the CXL bus is an asymmetric coherence bus using a master-slave (Master, Slave) mode, which does not allow direct interconnection between masters or slaves, limiting its application scenarios. Furthermore, other processors using the CXL bus controller require purchasing a license, which is expensive. The CCIX interconnect bus suffers from high latency, complex implementation, and low memory read performance; while NVLink is a proprietary protocol that other processors cannot use. In short, inter-processor interconnect buses are only designed for specific processors.

[0040] Server systems consist of diverse, heterogeneous processors and accelerators. Different processors use different interconnect buses, resulting in a lack of interoperability. Different bus protocols are also used between processors and between processors and I / O peripherals, requiring different processing methods. This current interconnect topology requires implementing multiple interconnect buses, making implementation complex; furthermore, memory cannot be shared between different types of servers, memory expansion access is limited, and the optimal performance of a diverse, heterogeneous distributed system cannot be achieved.

[0041] Therefore, this invention provides a multi-heterogeneous distributed system, a memory access method, and a storage medium. Each processor and accelerator has its own corresponding memory unit; a unified interconnect bus unit is deployed on each processor and accelerator; and a protocol adaptation interface module is deployed on the unified interconnect bus unit, enabling compatibility with various types of devices. By deploying a unified interconnect bus unit on each device, processors with different instruction sets, different types of accelerators, and different memory units can be unified into a single system, forming a supercomputer system. This enables memory sharing between different processors and between different processors and accelerators, forming a large memory pool, alleviating memory wall and I / O wall problems, and greatly improving the performance of the multi-heterogeneous distributed system. The unified interconnect bus unit can be used to interconnect different types of devices through different topologies, achieving consistent memory communication between devices within the system. It supports flexible interconnect topologies and can be flexibly expanded without affecting existing deployments.

[0042] Next, we will describe in detail a multi-element heterogeneous distributed system provided by the embodiments of the present invention. Figure 1 This is a schematic diagram of a multi-element heterogeneous distributed system provided in an embodiment of the present invention. The system includes at least two processors 11 and at least one accelerator 12. Each processor 11 and accelerator 12 has its own corresponding memory unit. A unified interconnect bus unit 13 is deployed on each processor 11 and accelerator 12. A protocol adaptation interface module is deployed on the unified interconnect bus unit 13. The protocol adaptation interface module is used to implement a conversion hub for the consistent protocol interface of different types of devices and the unified interconnect bus protocol interface.

[0043] Figure 1 The diagram below illustrates an example using two processors 11 and one accelerator 12. The actual number of processors 11 and accelerators 12 can be determined based on the actual business requirements of the heterogeneous distributed system, and is not limited here. The two processors 11 can be of the same type or different, and are not limited here.

[0044] In this embodiment of the invention, for ease of description, the processor 11 and the accelerator 12 can be collectively referred to as devices. The data processing methods between different devices are similar. The following is an explanation of any two devices among the processors 11 and the accelerators 12, namely the first device and the second device.

[0045] For ease of distinction, the Unified Cache Consistency Bus (UCCB) 13 deployed on the first device can be referred to as the first Unified Cache Consistency Bus 13, and the Unified Cache Consistency Bus 13 deployed on the second device can be referred to as the second Unified Cache Consistency Bus 13. Since the first Unified Cache Consistency Bus 13 and the second Unified Cache Consistency Bus 13 are only deployed on different devices, but have the same architecture, they are referred to by the same designation in this embodiment of the invention.

[0046] The first unified interconnect bus unit 13 is used to receive a read request sent by the first device through the protocol adaptation interface module, set the cache state of the first memory unit of the first device to an invalid state, encapsulate the read request into a request message that conforms to the set message format, and send the request message to the second unified interconnect bus unit 13 of the second device.

[0047] In practical applications, the first unified interconnect bus unit 13 can encapsulate the read request into a request message that conforms to the set message format according to the first identifier of the first device and the second identifier of the second device to which the read request points.

[0048] The second unified interconnect bus unit 13 is used to read data from the second memory unit of the second device after receiving a request message; set the cache state of the second memory unit to a shared state and encapsulate the data into a response message that conforms to the message format; and send the response message to the first unified interconnect bus unit 13.

[0049] In practical applications, the second unified interconnect bus unit 13 can read data from the second memory unit of the second device according to the second memory address carried in the request message.

[0050] The first unified interconnect bus unit 13 is used to store the data carried in the received response message into the first memory unit; and to set the cache state of the first memory unit to a shared state according to the cache state carried in the response message.

[0051] In practical applications, the unified interconnect bus unit 13 deployed on each processor 11 and accelerator 12 can be connected to the hub, thereby supporting different types of topologies.

[0052] The first unified interconnect bus unit 13 can send the request message to the second unified interconnect bus unit 13 of the second device through the hub; the second unified interconnect bus unit 13 can receive the request message through the hub and send the response message to the first unified interconnect bus unit 13 through the hub.

[0053] In this embodiment of the invention, the protocol adaptation interface module can adapt to the conformance protocol interfaces of different types of devices, and different processors have different external conformance protocol interfaces. By deploying the protocol adaptation interface module on the unified interconnect bus unit 13, the conversion between the conformance protocol interfaces of different types of devices and the unified interconnect bus protocol interface, i.e., the UCCB interface, can be realized.

[0054] Figure 2 This is a schematic diagram of the structure of a protocol adaptation interface module provided in an embodiment of the present invention. Figure 2 Taking the adaptation of three types of processors as an example, the protocol adaptation interface module can deploy three processor protocol conversion submodules, which can be referred to as the first processor protocol conversion submodule, the second processor protocol conversion submodule, and the third processor protocol conversion submodule for ease of description. Each processor protocol conversion submodule is used to adapt the conformance protocol interface of one type of processor. The three processor protocol conversion submodules are connected to a multiplexer (MUX / DEMUX) respectively.

[0055] In practical applications, by modifying the protocol adaptation interface module, such as by adding a device protocol conversion submodule, more types of devices can be adapted without modifying other modules.

[0056] Figure 3 This is a schematic diagram of an architecture for implementing cache-coherent interconnection between multiple types of processors and multiple types of accelerators, provided as an embodiment of the present invention. For ease of distinction, the multiple types of processors can be referred to as first-type processors, second-type processors, third-type processors, and fourth-type processors, respectively. The multiple types of accelerators can be referred to as first-type accelerators, second-type accelerators, and third-type accelerators, respectively. Figure 3 Each processor and each accelerator has its corresponding memory unit. In order to achieve interconnection between the devices, a unified interconnect bus unit is set on each processor and each accelerator. Figure 3 It also includes a dedicated storage medium for expanding the memory of a diverse, heterogeneous distributed system. A unified interconnect bus unit is also located on this dedicated storage medium, enabling interconnection with memory units on other devices.

[0057] In this embodiment of the invention, a unified interconnect bus unit 13 can be deployed on each device. The unified interconnect bus unit 13 may include a protocol layer, an adaptation layer, and a physical layer.

[0058] Figure 4This invention provides a block diagram of the internal functional modules of a unified interconnect bus unit. The unified interconnect bus unit may include a protocol layer, an adaptation layer, and a physical layer. The protocol layer may include a protocol adaptation interface module, a request queue management module, a response queue management module, a message parsing and encapsulation module, and a clock domain conversion module. The protocol adaptation interface module is connected to both the request queue management module and the response queue management module, and is used to transmit received requests to either the request queue management module or the response queue management module according to the request type. The message parsing and encapsulation module is connected to the request queue management module, the response queue management module, and the clock domain conversion module, and is used to encapsulate and decapsulate messages. The adaptation layer may include a protocol arbitration module, a connection state management module, a Cyclic Redundancy Check (CRC) control and retransmission module, and a bypass module. The physical layer may include a link control module, a local area network (LAN) emulation (Lane) path mapping and remapping module, a scrambling and descrambling module, and a bypass module.

[0059] The physical layer and adapter layer can be based on the open UCIe interconnect standard. The physical layer and adapter layer designed according to the UCIe interconnect standard have lower latency than PCIe-based interconnect buses such as CXL and CCIX, and are compatible with multiple protocols. Even if the protocol content is optimized later, it will not affect the physical layer and adapter layer. The physical layer can be used to implement functions such as link initialization and training. The adapter layer can be used to select and arbitrate between multiple protocols, and is also responsible for link state management. Bypass modules in the physical layer and adapter layer refer to the fact that these functional modules in both layers are optional.

[0060] In this embodiment of the invention, the bypass unit can be used to realize the direct connection between the adapter layer and the physical layer; when the first device and the second device are located on the same printed circuit board, the bypass units of the first adapter layer and the first physical layer of the first device and the second adapter layer and the second physical layer of the second device are in the open state to realize the direct connection between the first device and the second device.

[0061] In practical applications, when interconnecting homogeneous devices, i.e., interconnecting multiple devices on the same PCB motherboard, the link parameters can be statically configured because the link is known, simplifying the functions of the physical layer and the adaptation layer, basically achieving transparent transmission and minimizing latency.

[0062] During the initialization phase, the initialization of the physical layer and adaptation layer may include link training, protocol and parameter negotiation with remote nodes, discovery and enumeration of each node device, and configuration and querying of protocol registers. Protocol layer initialization may include configuring unique node IDs and configuring node forwarding tables to ensure arbitrary forwarding between nodes.

[0063] In this embodiment of the invention, for ease of distinction, the protocol adaptation interface module included in the protocol layer of the first unified interconnect bus unit 13 can be referred to as the first protocol adaptation interface module, the request queue management module as the first request queue management module, the response queue management module as the first response queue management module, the message parsing and encapsulation module as the first message parsing and encapsulation module, and the clock domain conversion module as the first clock domain conversion module. Similarly, the protocol adaptation interface module included in the protocol layer of the second unified interconnect bus unit 13 can be referred to as the second protocol adaptation interface module, the request queue management module as the second request queue management module, the response queue management module as the second response queue management module, the message parsing and encapsulation module as the second message parsing and encapsulation module, and the clock domain conversion module as the second clock domain conversion module.

[0064] Taking the first device reading data from the second device as an example, the first protocol adaptation interface module of the first unified interconnect bus unit 13 can be used to receive the read request sent by the first device and transmit the read request to the first request queue management module through the internal bus.

[0065] The first request queue management module is used to receive read requests; set the cache state of the first memory unit of the first device to an invalid state; query the address of the second device matching the second identifier carried by the second device in the node forwarding table; and transmit the read request and the address of the second device to the first message parsing and encapsulation module.

[0066] In this embodiment of the invention, a device can be considered a node, and the node forwarding table can record the correspondence between the identifiers and device addresses of each device in the multi-dimensional heterogeneous distributed system. The device identifier (ID) is unique and can be used to distinguish different devices.

[0067] The first message parsing and encapsulation module is used to encapsulate the received read request according to the message format to obtain the request message; and to transmit the request message and the second device address to the first clock domain conversion module.

[0068] The first clock domain conversion module is used to transmit the request message and the second device address to the hub through the first adaptation layer and the first physical layer. The hub is used to forward the request message to the second device corresponding to the second device address.

[0069] Besides acting as a relay as described above, the hub can also perform device identification functions. Taking the first device reading data from the second device as an example, the first protocol adaptation interface module of the first unified interconnect bus unit 13 is used to receive the read request sent by the first device and transmit the read request to the first request queue management module through the internal bus. The first request queue management module is used to receive the read request; set the cache state of the first memory unit of the first device to an invalid state; and transmit the read request to the first message parsing and encapsulation module. The first message parsing and encapsulation module is used to encapsulate the received read request according to the message format to obtain a request message; and transmit the request message to the first clock domain conversion module. The first clock domain conversion module is used to transmit the request message to the hub through the first adaptation layer and the first physical layer. The hub is used to look up the address of the second device matching the second identifier carried in the request message from the node forwarding table; and forward the request message to the second device corresponding to the second device address.

[0070] The message parsing and encapsulation module can contain a predefined message format. Table 1 is a schematic table of a message format provided in an embodiment of the present invention. The message in Table 1 contains eight parts, namely the identifier of the source device (source ID), the identifier of the destination device (destination ID), the message length (LEN), the message type (Type), the message sequence number (Tagid), the memory address (Addr), the memory data (Data), and the write enable signal (be).

[0071] Table 1

[0072] In a multi-node heterogeneous distributed system, each interconnected node has a unique ID number for routing, supporting a total of 256 nodes. Depending on the application requirements, message types can include read / write request messages, response messages, and listening messages. The message sequence number (Tagid) is used for packet loss handling such as retransmission. The memory address is 48 bits long. The memory data is the actual transmitted data; its write memory data width can be 256 bits or 512 bits. The write enable signal is byte-wise writable, with a width of data width / 8.

[0073] Taking the first device reading data from the second device as an example, the first message parsing and encapsulation module can encapsulate the first identifier of the first device, the second identifier of the second device to which the read request points, the message type and message sequence number to which the read request belongs, and the second memory address according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal, so as to obtain a request message; wherein, the second memory address may carry a status indicator that the first memory unit is in an invalid state.

[0074] The above description describes the operation flow of each module of the first unified interconnect bus unit 13 on the first device when the first device reads data from the second device. Next, the operation flow of each module on the second unified interconnect bus unit 13 will be described.

[0075] The second clock domain conversion module of the second unified interconnect bus unit 13 is used to receive request messages transmitted by the hub through the second physical layer and the second adaptation layer, and forward the request messages to the second message parsing and encapsulation module.

[0076] The second message parsing and encapsulation module is used to parse the request message to obtain the first identifier, message sequence number and second memory address of the first device; and send the second memory address to the second response queue management module.

[0077] The second response queue management module is used to read data from the second memory unit according to the second memory address and transmit the data to the second message parsing and encapsulation module; and to set the cache state of the second memory unit to a shared state.

[0078] The second message parsing and encapsulation module is used to receive data; according to the first identifier, it queries the node forwarding table for the first device address that matches the first identifier; it encapsulates the second identifier, the first identifier, the length of the data, the message type of the response request, the message sequence number, the first memory address, the data, and the write enable signal according to the message format to obtain a response message; and it transmits the response message and the first device address to the second clock domain conversion module; wherein, the first memory address carries a status identifier that the second memory unit is in a shared state.

[0079] The second clock domain conversion module is used to transmit the response message and the first device address to the hub through the second adaptation layer and the second physical layer. The hub is used to forward the response message to the first device corresponding to the first device address.

[0080] Besides acting as a relay as described above, the hub can also perform device identification functions. Taking the first device reading data from the second device as an example, the second clock domain conversion module of the second unified interconnect bus unit 13 is used to receive the request message transmitted by the hub through the second physical layer and the second adaptation layer, and forward the request message to the second message parsing and encapsulation module. The second message parsing and encapsulation module is used to parse the request message to obtain the first identifier, message sequence number and second memory address of the first device; and send the second memory address to the second response queue management module. The second response queue management module is used to read data from the second memory unit according to the second memory address, and transmit the data to the second message parsing and encapsulation module; the buffer state of the second memory unit is set to a shared state. The second message parsing and encapsulation module is used to receive data; query the node forwarding table for the first memory address matched by the first identifier; encapsulate the second identifier, the first identifier, the length of the data, the message type of the response request, the message sequence number, the first memory address, the data, and the write enable signal according to the message format to obtain a response message; and transmit the response message to the second clock domain conversion module; wherein the first memory address carries a status identifier indicating that the second memory unit is in a shared state. The second clock domain conversion module is used to transmit the response message to the hub through the second adaptation layer and the second physical layer; the hub is used to query the node forwarding table for the first device address matched by the first identifier carried in the response message; and forward the response message to the first device corresponding to the first device address.

[0081] Taking the example of the first device reading data from the second device, after the first device obtains the data transmitted from the second device, the first unified interconnect bus unit 13 can encapsulate the first identifier, the second identifier, the message type to which the listening belongs, and the message sequence number according to the message format to obtain the listening message; the listening message and the second memory address are then transmitted to the first clock domain conversion module. The first clock domain conversion module is used to transmit the listening message and the second device address to the hub through the first adaptation layer and the first physical layer; the hub is used to forward the listening message to the second device corresponding to the second device address, thus completing a complete read transaction request.

[0082] When sending a listening message, the hub acts not only as a relay but also as a device identification function. In practical applications, the first unified interconnect bus unit 13 can encapsulate the first identifier, the second identifier, the message type to which the listening belongs, and the message sequence number according to the message format to obtain a listening message; and transmit the listening message to the first clock domain conversion module. The first clock domain conversion module is used to transmit the listening message to the hub through the first adaptation layer and the first physical layer. The hub is used to look up the address of the second device matching the second identifier carried in the second device in the node forwarding table; and forward the listening message to the second device corresponding to the second device address.

[0083] Besides reading data from memory, common memory operations also include writing data to memory. The following section will use the example of a third device writing data to a first device to illustrate this.

[0084] For ease of distinction, the unified interconnect bus unit 13 deployed on the third device can be referred to as the third unified interconnect bus unit 13.

[0085] The third unified interconnect bus unit 13 can receive write requests sent by the third device, set the cache state of the third memory unit of the third device to an invalid state, encapsulate the data to be written into a write message that conforms to the message format, and send the write message to the first unified interconnect bus unit 13 through the hub.

[0086] The first unified interconnect bus unit 13 is used to receive write messages from the hub, store the data to be written carried in the write message into the first memory unit, set the cache state of the first memory unit to a unique clean state according to the cache state carried in the write message, and send monitoring messages to the second unified interconnect bus unit 13 through the hub. The second unified interconnect bus unit 13 is used to receive monitoring messages sent by the hub, and set the cache state of the second memory unit to an invalid state according to the invalid state carried in the monitoring message.

[0087] In this embodiment of the invention, there can be multiple categories of cache state. By setting the cache state, each device can intuitively understand the state of different memory units. For example, the device can determine whether the data on the memory unit is unique and has not been modified, or whether it is shared and has been modified.

[0088] In practical applications, "unique" can be used to indicate that data exists only in the current memory location and not in other memory locations. "Shared" indicates that data can exist in multiple memory locations. "Clean" indicates that the data has not been modified. "Dirty" indicates that the data has been modified.

[0089] In this embodiment of the invention, the unified interconnect bus unit 13 can determine the matching cache state according to the state of the data on its corresponding memory unit; wherein, the cache state may include invalid state, unique state and shared state; the unique state includes unique clean state, unique dirty state, unique clean free state and unique partially dirty state; the shared state includes shared clean state and shared dirty state.

[0090] The cache model defined by the Unified Interconnect Bus Unit supports different consistency protocols. Cache states include: Unique Clean (UC): The cache line is unique and "clean," existing only in the current cache and has not been modified. Unique Dirty (UD): The cache line is unique but "dirty," existing only in the current cache, but its data has been modified and not updated to memory. Shared Clean (SC): The cache line is not unique but "clean," copies may exist in other caches, and its data may have been modified, but it is "clean" in the current cache. Shared Dirty (SD): The cache line is not unique but "dirty," copies may exist in other caches, and its data has been modified. Invalid state: The cache line is invalid and not in the cache. Unique Clean Free (UCE): The cache line exists only in the current cache, is unique, but all data bytes are invalid. The unique part is dirty (UDP). The state of the cache line is unique but part of it is "dirty". The cache line only exists in the current cache. The cache line is unique, but only part of the data in the cache line is valid and "dirty".

[0091] Of the seven states mentioned above, the first five are for processors using the ACE (AXI Coherency Extensions) consistency interface, while the seven complete states are for processors using the CHI (Coherent Hub Interface) consistency interface. The queue management module internally determines which processor the consistency request originates from and selects which consistency states to use accordingly.

[0092] Before data is written, the first memory unit does not contain any data, so its cache state is invalid. After data is written to the first memory unit, since the data recorded in the first memory unit is the original data from the third memory unit, the cache state of the first memory unit can be set to a unique clean state.

[0093] In the above description, the first device reading data from the second device and the third device writing data to the first device are both examples of writing data to the same memory address of the first device. Therefore, after reading data from the second device and writing it to the memory location corresponding to that memory address, and then writing data to the same memory location of the first device by the third device, the memory address data of the second device is invalid. Therefore, the second unified interconnect bus unit 13 can set the cache state of the second memory unit to invalid state after receiving the monitoring message.

[0094] In this embodiment of the invention, for ease of distinction, the protocol adaptation interface module included in the protocol layer of the third unified interconnect bus unit 13 can be referred to as the third protocol adaptation interface module, the request queue management module as the third request queue management module, the response queue management module as the third response queue management module, the message parsing and encapsulation module as the third message parsing and encapsulation module, and the clock domain conversion module as the third clock domain conversion module.

[0095] The third protocol adaptation interface module of the third unified interconnect bus unit 13 receives the write request sent by the third device and transmits the write request to the third request queue management module through the internal bus.

[0096] The third request queue management module is used to receive write requests; set the cache status of the third memory unit of the third device to an invalid state; query the address of the first device matching the first identifier from the node forwarding table according to the first identifier of the first device carried in the write request; and transmit the write request and the address of the first device to the third message parsing and encapsulation module. The third message parsing and encapsulation module is used to encapsulate the received write request according to the message format to obtain a write message; and to transmit the write message and the first device address to the third clock domain conversion module. The third clock domain conversion module is used to transmit the write message and the first device address to the hub through the third adaptation layer and the third physical layer; the hub is used to forward the write message to the first device corresponding to the first device address.

[0097] Besides acting as a relay as described above, the hub can also perform device identification functions. Taking a third device writing data to a first device as an example, the third protocol adaptation interface module of the third unified interconnect bus unit 13 is used to receive the write request sent by the third device and transmit the write request to the third request queue management module through the internal bus. The third request queue management module is used to receive the write request; set the cache state of the third memory unit of the third device to an invalid state; look up the first device address matching the first identifier carried by the first device in the node forwarding table; and transmit the write request and the first device address to the third message parsing and encapsulation module. The third message parsing and encapsulation module is used to encapsulate the received write request according to the message format to obtain a write message; and transmit the write message to the third clock domain conversion module; the third clock domain conversion module is used to transmit the write message to the hub through the third adaptation layer and the third physical layer. The hub is used to look up the first device address matching the first identifier carried by the first device in the node forwarding table; and forward the write message to the first device corresponding to the first device address.

[0098] In the specific implementation, the third message parsing and encapsulation module can encapsulate the third identifier of the third device, the first identifier of the first device to which the write request points, the message type and message sequence number to which the write request belongs, the first memory address, and the data to be written according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal, so as to obtain a write message; wherein, the first memory address carries a status indicator that the third memory unit is in an invalid state.

[0099] In this embodiment of the invention, each processor 11 may include domestically produced processors with different instruction sets; the accelerator 12 includes a graphics processor and a field-programmable gate array.

[0100] There are two main types of domestically produced processors. One type uses the open-source and free Advanced Micro Devices (AMD) and Reduced Instruction Set Computer (RISC-V) architectures. The other type uses the open but licensed ARM architecture. Accelerators can utilize domestically produced accelerators, such as domestically produced graphics processing units (GPUs) and domestically produced field-programmable gate arrays (FPGAs).

[0101] When the first device is an ARM processor 11, the first device can send a read request to the first unified interconnect bus unit 13 via the CHI interface. When the first device is a RISC-V processor 11, the first device can send a read request to the first unified interconnect bus unit 13 via the ACE interface.

[0102] To expand memory space, in addition to the memory units corresponding to each processor 11 and accelerator 12, storage media independent of those of each processor 11 and accelerator 12 can be configured; a unified interconnect bus unit 13 is deployed on the storage media. The storage media can be persistent memory (Intel Optane Persistent Memory, PMEM), low-power double data rate synchronous dynamic random access memory (LPDDR), solid state drive (SSD), etc.

[0103] This section uses a server system with multiple processors and heterogeneous accelerators as an example to illustrate how reading and writing data from memory can be performed. Taking the example of a first-type processor reading data from a second-type processor, the operation process can include: (1) When the first type of processor wants to read the memory data of the second type of processor, the first type of processor sends a read request to the UCCB through the CHI interface. After the UCCB protocol adaptation interface module processes the read request, it sends it to the request queue management module through the internal bus. The cache state of the internal cache state machine is set to invalid state. The message parsing and encapsulation module encapsulates the read request into a request message according to the message format corresponding to Table 1. The source ID of the request message is the ID of the first type of processor, the destination ID points to the second type of processor, the Type is a unique read transaction request, the Tagid is 1, the address is the memory address of the second type of processor, and the data is 0. After passing through the adaptation layer and the physical layer, it is sent to the hub, and the hub directly forwards it to the second type of processor.

[0104] (2) When the UCCB of the second type processor receives the read message, it changes the internal cache state to shared clean (SC), reads the data from memory and assembles it into a response message of the corresponding message format. The destination ID of the response message is the ID of the first type processor, the type is response acknowledgment, the tagid is the same as the read request and is still 1, the addr is the memory address of the first type processor, the data is the memory data, and it is sent to the hub. The hub forwards it to the first type processor.

[0105] (3) After receiving the response message, the UCCB on the first type of processor updates the cache status to SC and sends a listening message to the second type of processor. The source ID of the listening message is the ID of the first type of processor, the destination ID points to the second type of processor, the type is listening response, and the address and data are both 0. This completes one read transaction request.

[0106] Based on the first type of processor reading data from the second type of processor, taking the example of a domestically produced graphics processor writing data to the same memory address of the first type of processor, the operation process can include: (4) When a domestically produced graphics processor (GPU) prepares to write data to the same memory address as a first-class processor, its UCCB initiates a write transaction request, encapsulates it into a write message, and sets the source ID of the write message to the GPU ID, the destination ID to the first-class processor ID, the Type to the write transaction request, the Tagid to 1, the Tddr to the write request address (which is the same as the read request address), and the Data to the memory data to be written. The packet is then sent to the hub, which sends the write message to the first-class processor based on the device address corresponding to the destination ID. The GPU's UCCB sets the cache state to invalid. In this embodiment of the invention, writing and reading are different operations, and the Type can distinguish between them. Therefore, the Tagid can be represented by the same number for both write and read operations. The Tagid can start from 1 and increment for requests with the same destination ID.

[0107] (5) The UCCB of the first type of processor receives the write message. After the message parsing and encapsulation module parses it, it stores the data in the memory at the corresponding address and sets the cache status to unique clean (UC). At the same time, it returns a write response message (with the same format as the read response) to the domestic graphics processor.

[0108] (6) The memory address data of the second type of processor is invalid. The UCCB of the first type of processor initiates a monitoring message to inform the second type of processor of the invalid status. In the message format, Type is the monitoring request, Address is the write request address, and Data is the invalid status indication.

[0109] (7) The UCCB of the second type of processor receives the monitoring request and invalidates its internal cache state. Thus, a consistent read request and a consistent write request are completed.

[0110] The server system can include domestically produced processors with different instruction sets and various types of domestically produced accelerators. Both the domestic processors and accelerators include corresponding domestically produced memory units. In practical applications, the server system can also be configured with independent storage media, which can utilize domestically produced PMEM. The devices are interconnected via a UCCB interconnect bus, where the interconnect topology supports fully interconnected mesh topologies, crossbar topologies, etc., enabling interconnection between processors, between accelerators, and between processors and accelerators, supporting system-wide sharing of memory resources. Under this topology, any device supporting UCCB can be added.

[0111] The UCCB set by this invention can unify processors with different instruction sets, different types of accelerators, and different memory units into one system to form a supercomputer system. It can realize the sharing of local memory on the processor and remote memory on the accelerator, forming a large memory pool, while alleviating the memory wall and I / O wall.

[0112] As can be seen from the above technical solution, the multi-heterogeneous distributed system includes at least two processors and at least one accelerator; each processor and accelerator has its own corresponding memory unit; each processor and accelerator is equipped with a unified interconnect bus unit; a protocol adaptation interface module is deployed on the unified interconnect bus unit; the protocol adaptation interface module is used to convert the consistency protocol interface of different types of devices to the unified interconnect bus protocol interface. The first device and the second device are any two devices among the processors and accelerators. The first unified interconnect bus unit deployed on the first device is used to receive read requests sent by the first device through the protocol adaptation interface module. Since there is currently no data in the first memory unit, the cache state of the first memory unit of the first device can be set to an invalid state first; the read request is encapsulated into a request message conforming to the set message format and sent to the second unified interconnect bus unit of the second device. The second unified interconnect bus unit is used to read data from the second memory unit of the second device after receiving the request message; since data sharing is required, the cache state of the second memory unit can be set to a shared state, and the data is encapsulated into a response message conforming to the message format; the response message is sent to the first unified interconnect bus unit. The first unified interconnect bus unit is used to store the data carried in the received response message into the first memory unit; and to set the cache state of the first memory unit to a shared state according to the cache state carried in the response message. The beneficial effect of this invention is that by deploying a unified interconnect bus unit on each device and deploying a protocol adapter interface module on the unified interconnect bus unit, compatibility with various types of devices can be achieved. This allows processors with different instruction sets, different types of accelerators, and different memory units to be unified into a single system, forming a supercomputer system. This enables memory sharing between different processors and between different processors and accelerators, forming a large memory pool and alleviating the memory wall and I / O wall problems. The unified interconnect bus unit can be used to interconnect different types of devices through different topologies, supporting flexible interconnect topologies and allowing for flexible scaling without affecting existing deployments. By recording the cache state of the memory unit, consistent memory communication between devices within the system is achieved. The deployment method of this invention greatly improves the performance of a multi-dimensional heterogeneous distributed system.

[0113] Figure 5 A flowchart of a memory access method provided in an embodiment of the present invention includes: S501: The protocol adaptation interface module receives a read request sent by the first device and sets the cache state of the first memory unit of the first device to an invalid state.

[0114] The protocol adaptation interface module is used to convert between the consistent protocol interface of different types of devices and the unified interconnect bus protocol interface.

[0115] S502: Based on the first identifier of the first device and the second identifier of the second device to which the read request points, the read request is encapsulated into a first request message conforming to the set message format.

[0116] S503: Send the first request message to the second unified interconnect bus unit of the second device.

[0117] S504: Receives the response message from the second unified interconnect bus unit and stores the data carried in the response message in the first memory unit.

[0118] S505: Based on the cache status carried in the response message, set the cache status of the first memory unit to the shared state.

[0119] In some embodiments, it also includes: Receive a second request message sent by the hub; wherein the second request message is transmitted to the hub by the second unified interconnect bus unit of the second device; Data is read from the first memory unit based on the first memory address carried in the second request message; Set the cache state of the first memory unit to the shared state and encapsulate the data into a response message that conforms to the message format; The response message is sent to the second device via the hub.

[0120] In some embodiments, it also includes: The device receives a write message from the hub and stores the data to be written carried in the write message into the first memory unit; wherein the write message is transmitted to the hub by the third unified interconnect bus unit of the third device. Based on the cache state carried in the write message, set the cache state of the first memory unit to a unique clean state; The hub sends monitoring messages to the second unified interconnect bus unit so that the second unified interconnect bus unit can receive the monitoring messages sent by the hub; based on the invalid status carried in the monitoring message, the cache status of the second memory unit is set to invalid status.

[0121] Figure 5 For a description of the features in the corresponding embodiments, please refer to Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0122] As can be seen from the above technical solution, the unified interconnect bus unit deployed on the first device can receive read requests sent by the first device based on the protocol adaptation interface module, and set the cache state of the first memory unit of the first device to an invalid state. Based on the first identifier of the first device and the second identifier of the second device to which the read request points, the read request is encapsulated into a first request message conforming to a set message format. The first request message is sent to the second unified interconnect bus unit of the second device, so that the second unified interconnect bus unit can read data from the second memory unit of the second device according to the second memory address carried in the request message; set the cache state of the second memory unit to a shared state, and encapsulate the data into a response message conforming to the message format; the response message is sent to the first unified interconnect bus unit. The first unified interconnect bus unit receives the response message fed back by the second unified interconnect bus unit, and stores the data carried in the response message into the first memory unit. Based on the cache state carried in the response message, the cache state of the first memory unit is set to a shared state. The beneficial effects of this invention lie in the fact that by deploying a unified interconnect bus unit on each device, and then deploying protocol adapter interface modules on that unified interconnect bus unit, compatibility with various types of devices can be achieved. This allows for the unification of processors with different instruction sets, different types of accelerators, and different memory units into a single system, forming a supercomputer system. It enables memory sharing between different processors and between different processors and accelerators, creating a large memory pool and alleviating memory wall and I / O wall problems. The unified interconnect bus unit can be used to interconnect different types of devices through different topologies, supporting flexible interconnect topologies and allowing for flexible scaling without affecting existing deployments. By recording the cache state of memory units, consistent memory communication between devices within the system is achieved. The deployment method of this invention significantly improves the performance of a diverse, heterogeneous distributed system.

[0123] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the memory access method described above.

[0124] The foregoing has provided a detailed description of a multi-element heterogeneous distributed system, a memory access method, and a computer-readable storage medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0125] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] The foregoing has provided a detailed description of a multi-element heterogeneous distributed system, a memory access method, and a computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A multi-element heterogeneous distributed system, characterized in that, It includes at least two processors and at least one accelerator; wherein each processor and accelerator has its own corresponding memory unit; each processor and accelerator is equipped with a unified interconnect bus unit, and a protocol adaptation interface module is deployed on the unified interconnect bus unit; the protocol adaptation interface module is used to realize the conversion between the consistent protocol interface of different types of devices and the unified interconnect bus protocol interface; The first unified interconnect bus unit is configured to receive a read request sent by the first device through the protocol adaptation interface module, set the cache state of the first memory unit of the first device to an invalid state, encapsulate the read request into a request message conforming to a set message format and send it to the second unified interconnect bus unit of the second device; wherein the first device and the second device are any two devices among the processors and the accelerators; The second unified interconnect bus unit is configured to, upon receiving the request message, read data from the second memory unit of the second device; set the cache state of the second memory unit to a shared state; encapsulate the data into a response message conforming to the message format; and send the response message to the first unified interconnect bus unit. The first unified interconnect bus unit is used to store the data carried in the received response message into the first memory unit; and to set the cache state of the first memory unit to a shared state according to the cache state carried in the response message. The unified interconnect bus unit includes a protocol layer, an adaptation layer, and a physical layer. The protocol layer includes a protocol adaptation interface module, a request queue management module, a response queue management module, a message parsing and encapsulation module, and a clock domain conversion module. The protocol adaptation interface module is connected to both the request queue management module and the response queue management module, and is used to transmit received requests to either the request queue management module or the response queue management module according to the request type. The message parsing and encapsulation module is connected to the request queue management module, the response queue management module, and the clock domain conversion module, and is used to encapsulate and decapsulate messages. The first unified interconnect bus unit includes a first message parsing and encapsulation module; the first message parsing and encapsulation module is used to encapsulate the received read request according to the message format to obtain a request message; The first message parsing and encapsulation module is used to encapsulate the first identifier of the first device, the second identifier of the second device to which the read request points, the message type and message sequence number to which the read request belongs, and the second memory address according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal, so as to obtain the request message; wherein, the second memory address carries a status indicator that the first memory unit is in an invalid state.

2. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The first protocol adaptation interface module of the first unified interconnect bus unit is used to receive the read request sent by the first device and transmit the read request to the first request queue management module through the internal bus; The first request queue management module is used to receive the read request; Set the cache state of the first memory unit of the first device to an invalid state; Based on the second identifier of the second device carried in the read request, the address of the second device matching the second identifier is queried from the node forwarding table; the read request and the address of the second device are then transmitted to the first packet parsing and encapsulation module. The first message parsing and encapsulation module is used to transmit the request message and the second device address to the first clock domain conversion module; The first clock domain conversion module is used to transmit the request message and the second device address to the hub through the first adaptation layer and the first physical layer; The hub is used to forward the request message to the second device corresponding to the second device address.

3. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The first protocol adaptation interface module of the first unified interconnect bus unit is used to receive the read request sent by the first device and transmit the read request to the first request queue management module through the internal bus; The first request queue management module is used to receive the read request; Set the cache state of the first memory unit of the first device to an invalid state; The read request is transmitted to the first message parsing and encapsulation module; The first message parsing and encapsulation module is used to transmit the request message to the first clock domain conversion module; The first clock domain conversion module is used to transmit the request message to the hub through the first adaptation layer and the first physical layer; The hub is configured to query the address of the second device that matches the second identifier from the node forwarding table based on the second identifier of the second device carried in the request message; and forward the request message to the second device corresponding to the second device address.

4. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The second clock domain conversion module of the second unified interconnect bus unit is used to receive the request message transmitted by the hub through the second physical layer and the second adaptation layer, and forward the request message to the second message parsing and encapsulation module; The second message parsing and encapsulation module is used to parse the request message to obtain the first identifier of the first device, the message sequence number, and the second memory address; Send the second memory address to the second response queue management module; The second response queue management module is used to read data from the second memory unit according to the second memory address, and transmit the data to the second message parsing and encapsulation module; Set the cache state of the second memory unit to a shared state; The second message parsing and encapsulation module is used to receive the data; query the node forwarding table for the first device address matched by the first identifier according to the first identifier; encapsulate the second identifier, the first identifier, the length of the data, the message type of the response request, the message sequence number, the first memory address, the data, and the write enable signal according to the message format to obtain the response message; and transmit the response message and the first device address to the second clock domain conversion module; wherein, the first memory address carries a status identifier indicating that the second memory unit is in a shared state. The second clock domain conversion module is used to transmit the response message and the first device address to the hub through the second adaptation layer and the second physical layer; The hub is used to forward the response message to the first device corresponding to the first device address.

5. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The second clock domain conversion module of the second unified interconnect bus unit is used to receive the request message transmitted by the hub through the second physical layer and the second adaptation layer, and forward the request message to the second message parsing and encapsulation module; The second message parsing and encapsulation module is used to parse the request message to obtain the first identifier of the first device, the message sequence number, and the second memory address; Send the second memory address to the second response queue management module; The second response queue management module is used to read data from the second memory unit according to the second memory address and transmit the data to the second message parsing and encapsulation module; the cache state of the second memory unit is set to a shared state; The second message parsing and encapsulation module is used to receive the data; Based on the first identifier, the first memory address matching the first identifier is queried from the node forwarding table; the second identifier, the first identifier, the length of the data, the message type of the response request, the message sequence number, the first memory address, the data, and the write enable signal are encapsulated according to the message format to obtain the response message; the response message is transmitted to the second clock domain conversion module; wherein, the first memory address carries a status identifier indicating that the second memory unit is in a shared state; The second clock domain conversion module is used to transmit the response message to the hub through the second adaptation layer and the second physical layer; The hub is configured to query the first device address matching the first identifier from the node forwarding table based on the first identifier carried in the response message; and forward the response message to the first device corresponding to the first device address.

6. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The first unified interconnect bus unit is used to encapsulate the first identifier, the second identifier, the message type to which the monitoring belongs, and the message sequence number according to the message format to obtain a monitoring message; The listening message and the second memory address are transmitted to the first clock domain conversion module; The first clock domain conversion module is used to transmit the listening message and the second device address to the hub through the first adaptation layer and the first physical layer; The hub is used to forward the listening message to the second device corresponding to the second device address.

7. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The first unified interconnect bus unit is used to encapsulate the first identifier, the second identifier, the message type to which the monitoring belongs, and the message sequence number according to the message format to obtain a monitoring message; The monitored message is transmitted to the first clock domain conversion module; The first clock domain conversion module is used to transmit the monitoring message to the hub through the first adaptation layer and the first physical layer; The hub is configured to query the address of the second device that matches the second identifier from the node forwarding table based on the second identifier of the second device carried in the monitoring message; and forward the monitoring message to the second device corresponding to the second device address.

8. The multi-element heterogeneous distributed system according to claim 1, characterized in that, The third unified interconnect bus unit is used to receive write requests sent by the third device, set the cache state of the third memory unit of the third device to an invalid state, and encapsulate the data to be written into a write message that conforms to the message format. The write message is sent to the first unified interconnect bus unit via the hub; The first unified interconnect bus unit is used to receive the write message fed back by the hub and store the data to be written carried in the write message into the first memory unit. Based on the cache state carried in the write message, the cache state of the first memory unit is set to a unique clean state; The hub sends a monitoring message to the second unified interconnect bus unit. The second unified interconnect bus unit is used to receive the monitoring message sent by the hub; Based on the invalid status carried in the monitoring message, the cache status of the second memory unit is set to invalid.

9. The multi-element heterogeneous distributed system according to claim 8, characterized in that, The third protocol adaptation interface module of the third unified interconnect bus unit is used to receive the write request sent by the third device and transmit the write request to the third request queue management module through the internal bus. The third request queue management module is used to receive the write request; Set the cache state of the third memory unit of the third device to an invalid state; Based on the first identifier of the first device carried in the write request, the first device address matching the first identifier is queried from the node forwarding table; the write request and the first device address are then transmitted to the third packet parsing and encapsulation module. The third message parsing and encapsulation module is used to encapsulate the received write request according to the message format to obtain the write message; and to transmit the write message and the first device address to the third clock domain conversion module. The third clock domain conversion module is used to transmit the write message and the first device address to the hub through the third adaptation layer and the third physical layer; The hub is used to forward the write message to the first device corresponding to the first device address.

10. The multi-element heterogeneous distributed system according to claim 8, characterized in that, The third protocol adaptation interface module of the third unified interconnect bus unit is used to receive the write request sent by the third device and transmit the write request to the third request queue management module through the internal bus. The third request queue management module is used to receive the write request; Set the cache state of the third memory unit of the third device to an invalid state; Based on the first identifier of the first device carried in the write request, the first device address matching the first identifier is queried from the node forwarding table; the write request and the first device address are then transmitted to the third packet parsing and encapsulation module. The third message parsing and encapsulation module is used to encapsulate the received write request according to the message format to obtain the write message; and to transmit the write message to the third clock domain conversion module. The third clock domain conversion module is used to transmit the write message to the hub through the third adaptation layer and the third physical layer; The hub is used to query the address of the first device that matches the first identifier from the node forwarding table based on the first identifier of the first device carried in the write message; The write message is forwarded to the first device corresponding to the first device address.

11. The multi-element heterogeneous distributed system according to claim 9 or 10, characterized in that, The third message parsing and encapsulation module is used to encapsulate the third identifier of the third device, the first identifier of the first device to which the write request points, the message type and message sequence number to which the write request belongs, the first memory address, and the data to be written according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal, so as to obtain the write message; wherein, the first memory address carries a status indicator that the third memory unit is in an invalid state.

12. The multi-element heterogeneous distributed system according to claim 1, characterized in that, Both the adapter layer and the physical layer are provided with bypass units; wherein, the bypass units are used to realize direct connection between the adapter layer and the physical layer; When the first device and the second device are located on the same printed circuit board, the bypass units of the first adapter layer and the first physical layer of the first device and the second adapter layer and the second physical layer of the second device are in the open state to realize the direct connection between the first device and the second device.

13. The multi-element heterogeneous distributed system according to claim 1, characterized in that, It also includes a storage medium independent of each of the processors and the accelerators; the unified interconnect bus unit is deployed on the storage medium.

14. The multi-element heterogeneous distributed system according to claim 1, characterized in that, Each of the unified interconnect bus units is used to determine the matching cache state based on the state of the data on its corresponding memory unit; wherein, the cache state includes invalid state, unique state and shared state; the unique state includes unique clean state, unique dirty state, unique clean free state and unique partially dirty state; the shared state includes shared clean state and shared dirty state.

15. A memory access method, characterized in that, include: The protocol adaptation interface module receives a read request from the first device and sets the cache state of the first memory unit of the first device to an invalid state. The protocol adaptation interface module is used to convert between the consistency protocol interface of different types of devices and the first unified interconnect bus protocol interface. The unified interconnect bus unit deployed on the first device is a first unified interconnect bus unit. The unified interconnect bus unit includes a protocol layer, an adaptation layer, and a physical layer. The protocol layer includes the protocol adaptation interface module, a request queue management module, a response queue management module, a message parsing and encapsulation module, and a clock domain conversion module. The protocol adaptation interface module is connected to both the request queue management module and the response queue management module, and is used to transmit the received request to the request queue management module or the response queue management module according to the request type. The message parsing and encapsulation module... The modules are connected to the request queue management module, the response queue management module, and the clock domain conversion module, respectively, and are used to encapsulate and decapsulate messages. The first unified interconnect bus unit includes a first message parsing and encapsulation module. The first message parsing and encapsulation module is used to encapsulate the received read request according to the message format to obtain a request message. The first message parsing and encapsulation module is used to encapsulate the first identifier of the first device, the second identifier of the second device pointed to by the read request, the message type and message sequence number of the read request, and the second memory address according to the message format of the source device identifier, the destination device identifier, the message length, the message type, the message sequence number, the memory address, the memory data, and the write enable signal to obtain the request message. The second memory address carries a status indicator that the first memory unit is in an invalid state. Based on the first identifier of the first device and the second identifier of the second device to which the read request points, the read request is encapsulated into a first request message conforming to a set message format; The first request message is sent to the second unified interconnect bus unit of the second device; Receive the response message from the second unified interconnect bus unit and store the data carried in the response message in the first memory unit; Based on the cache status carried in the response message, the cache status of the first memory unit is set to a shared state.

16. The memory access method according to claim 15, characterized in that, Also includes: Receive a second request message sent by the hub; wherein the second request message is transmitted to the hub by the second unified interconnect bus unit of the second device; Data is read from the first memory unit according to the first memory address carried in the second request message; Set the cache state of the first memory unit to a shared state, and encapsulate the data into a response message conforming to the message format; The response message is sent to the second device via the hub.

17. The memory access method according to claim 15, characterized in that, Also includes: The device receives a write message from the hub and stores the data to be written carried in the write message into the first memory unit; wherein the write message is transmitted to the hub by the third unified interconnect bus unit of the third device. Based on the cache state carried in the write message, the cache state of the first memory unit is set to a unique clean state; The hub sends a monitoring message to the second unified interconnect bus unit so that the second unified interconnect bus unit can receive the monitoring message sent by the hub; and sets the cache state of the second memory unit to invalid state according to the invalid state carried in the monitoring message.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the memory access method as described in any one of claims 15 to 17.

Images