Resource decoupling system applied to data center and response method
By introducing a dedicated interconnect controller and management module into the resource pool and using a broadcast packet mechanism for resource detection and address mapping, the problem of high network communication latency in the resource decoupling architecture is solved, enabling efficient access and flexible management between resource entities.
Patent Information
- Application Number
- CN202511485506.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-17
- Publication Date
- 2026-01-09
AI Technical Summary
In the existing resource decoupling architecture, the network communication latency between resource pools is high, and it is necessary to reduce the access latency between different resource entities.
A dedicated interconnect controller and management module are introduced to perform resource detection through broadcast packets with a set format, dynamically maintain an independent address mapping table, realize direct network communication between resource entities, and reduce the involvement of CPU and memory.
It reduces the latency of mutual access between resource entities, improves the system's ability to flexibly manage heterogeneous resources, simplifies operation and maintenance complexity, and adapts to the data center's dynamic allocation needs for heterogeneous resources.
Smart Images
Figure CN121309536A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of data center management, resource request and response technology, and in particular to a resource decoupling system and response method applied to data centers. Background Technology
[0002] Based on resource decoupling, the basic deployment unit of a data center is no longer a complete machine node, but rather a collection of different hardware units in multiple resource pools. In this architecture, data center operators can manage and replace each resource independently.
[0003] Existing resource decoupling architectures mostly consist of resource pools containing different computing or storage resources, such as SSD resource pools, NVM resource pools, GPU resource pools, and NPU resource pools, along with an interconnecting network. Each resource pool contains at least one set of CPUs, memory modules, and network interface cards (NICs) responsible for network communication between the resource pools. When different resource pools need to exchange data, the data is first moved to the memory of the current resource pool. Then, the CPU initiates network communication. Once the data reaches the destination resource pool, the CPU of the destination resource writes the data to its storage space. For example, when a GPU resource pool wants to access an SSD resource pool, the data that the GPU needs to write is first copied from its GPU memory to the CPU memory, and then the CPU is notified to initiate network communication. The CPU initiates network communication, sending the data from its own memory to the CPU memory in the SSD resource pool, and then the CPU in the SSD resource pool copies the data from its memory to the SSD storage resource.
[0004] In this system architecture, the data and control paths for network communication between different resource pools must be handled by the CPU of that resource pool, increasing communication latency. A core requirement of a resource-decoupled architecture is to minimize network communication latency. Therefore, improvements are needed to the existing architecture and resource access methods to avoid the involvement of CPU and memory, thereby reducing latency in inter-resource access. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a resource decoupling system and response method for data centers, which can realize network-centric resource decoupling and reduce the latency of mutual access between different resource entities.
[0006] The technical solution adopted by this invention to solve its technical problem is as follows: A resource decoupling system applied to a data center is provided, including multiple resource pools and management modules of different resource types. Each resource pool includes several resource entities and an interconnect controller that provides data communication services for the resource entities. The management module is communicatively connected to each interconnect controller and is used to perform resource probing of each resource entity using broadcast packets of a set format. It creates an independent address mapping table for each resource type and dynamically maintains it based on the resource probing results. Simultaneously, it coordinates mutual access between different resource entities through commands of a set format. Each resource entity's interconnect controller caches the address mapping table corresponding to the resources it has accessed locally to accelerate the address lookup process for subsequent resource requests.
[0007] Furthermore, the format of the broadcast packet includes packet type, resource type, and resource capacity.
[0008] Furthermore, the management module uses the following methods for resource detection:
[0009] Send the first broadcast packet, whose packet type is "probe" and all others are empty;
[0010] Receive a second broadcast packet sent by a resource entity. The packet type of the second broadcast packet is "response", "declare", or "remove". The resource type and resource capacity of the second broadcast packet are filled in with the corresponding resource entity information.
[0011] The resource type and resource capacity of the corresponding resource entity are determined based on the received second broadcast packet.
[0012] Furthermore, the dynamic maintenance based on resource detection results includes:
[0013] If a second broadcast packet of type "response" or "declaration" is received, then according to the detected resource type and resource capacity, a global address to network address mapping is assigned to the corresponding resource entity, and the obtained mapping relationship is added to the address mapping table of the corresponding resource type;
[0014] If a second broadcast packet of type "remove" is received during resource probing, the mapping relationship of the corresponding resource entity is deleted from the address mapping table of the corresponding resource type.
[0015] Furthermore, the entries in the address mapping table include a start address, an end address, and a network address.
[0016] Furthermore, adding the obtained mapping relationship to the address mapping table of the corresponding resource type includes:
[0017] Use the address mapping table of the resource type corresponding to the detected resource entity as the target address mapping table;
[0018] Fill the starting address of the target address mapping table with the current starting address, calculate the sum of the current starting address of the target address mapping table and the resource capacity and fill it with the ending address, fill the network address of the resource entity with the network address, generate a new table entry and add it to the target address mapping table;
[0019] Update the starting address of the target address mapping table to the address following the calculated ending address.
[0020] Furthermore, deleting the mapping relationship of the corresponding resource entity from the address mapping table of the corresponding resource type includes:
[0021] Using the network address as an index, search for and remove the entry corresponding to the detected resource entity in the target address mapping table.
[0022] Furthermore, the management module is also used to coordinate mutual access between different resource entities through commands with defined formats.
[0023] The present invention also provides a resource request and response method, applied to the system described above, comprising the following steps:
[0024] The first resource entity issues a resource request command. The command format includes command type, resource type, local resource address, destination resource address, and resource length. The command type includes "write request" and "read request".
[0025] In response to a resource request command, the interconnect controller of the first resource entity looks up the mapping from the destination resource address to the network address in the local address mapping table. If a match is found, it interacts with the interconnect controller of the second resource entity based on the command type. If no match is found, it encapsulates the resource request command into a first data packet and sends it.
[0026] The management module parses the first data packet received, locates the target address mapping table according to the parsed resource type, and looks up the network address of the target resource. If no match is found, the interconnect controller returns an error code to the first resource entity and terminates the resource request and response process. If a match is found, the matched entry is returned to the interconnect controller of the first resource entity.
[0027] The interconnect controller of the first resource entity writes the entries returned by the management module into the local address mapping table and interacts with the interconnect controller of the second resource entity according to the command type.
[0028] Furthermore, the data interaction with the interconnection controller of the second resource entity based on the command type includes:
[0029] If the command type is "write request":
[0030] The interconnection controller of the first resource entity reads the target write data according to the local resource address and resource length, and encapsulates it together with the command type, source network address, destination network address, destination start address and resource length into a second data packet for transmission.
[0031] The interconnection controller of the second resource entity parses the received second data packet, extracts the target write data, and writes it into the corresponding address space of the second resource entity;
[0032] If the command type is "read request":
[0033] The interconnection controller of the first resource entity encapsulates the command type, network address, starting address, and resource length into a second data packet for transmission.
[0034] The interconnect controller of the second resource entity parses the received second data packet, reads the target read data according to the local resource address and resource length, and encapsulates it into a third data packet and sends it to the interconnect controller of the first resource entity.
[0035] The interconnect controller of the first resource entity extracts the target read data from the third data packet and writes it to the resource entity.
[0036] Beneficial effects
[0037] By adopting the above-mentioned technical solutions, this invention has the following advantages and positive effects compared with the prior art: This invention, by introducing a dedicated interconnect controller into the resource pool and combining it with an independent address mapping table for each resource type dynamically maintained by the management module, can replace the CPU's takeover of data and control paths in the traditional architecture, directly realizing network communication between resource entities. This eliminates the overhead of data movement between the CPU, memory, and resource entities, thereby reducing the latency of mutual access between different resources. This invention, through a broadcast packet mechanism, realizes resource detection including "response," "report," and "removal," supporting dynamic access, removal, and address mapping updates of resource entities, improving the system's flexible management capabilities for heterogeneous resources. The address mapping table dynamically allocates a global address range based on resource capacity, ensuring efficient use of address space and conflict avoidance. The basic deployment unit of the data center is decoupled from the entire machine node into an independent resource pool. Each resource can be managed, upgraded, or replaced independently, reducing hardware coupling and adapting to the data center's need for dynamic allocation of heterogeneous resources, while simplifying operation and maintenance complexity. Attached Figure Description
[0038] Figure 1 This is a system schematic diagram of the first embodiment of the present invention;
[0039] Figure 2 This is a system schematic diagram of a preferred embodiment 1 of the first embodiment of the present invention;
[0040] Figure 3 This is a flowchart of the second embodiment of the present invention. Detailed Implementation
[0041] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
[0042] The first embodiment of the present invention relates to a resource decoupling system applied to a data center, such as... Figure 1 As shown, the system includes multiple resource pools and a management module with different resource types. Each resource pool comprises several resource entities and an interconnect controller that provides data communication services to these entities. The management module communicates with each interconnect controller and uses broadcast packets of a defined format to probe the resources of each entity. It creates an independent address mapping table for each resource type and dynamically maintains it based on the resource probe results. Each resource entity's interconnect controller caches a portion of the management module's address mapping table to accelerate the address lookup process during resource requests.
[0043] Resource entities can communicate with the interconnect controller via the system bus. Their function is to provide certain computing and storage capabilities and to communicate with other resource entities via the network. Resource entities can be central processing units (CPUs), graphics processing units (GPUs), tensor processors (TPUs), neural network processors (NPUs), memory, SSDs, etc. Based on communication requirements, resource entities can be divided into two categories: Master and Slave. Master-type resource entities can both access other resource entities and be accessed by other resource entities; Slave-type resource entities can only be accessed by other resource entities. Generally, Master-type resource entities have certain computing capabilities and may need to exchange data with other Masters or read and write data to Slave resources during computation. Slave-type resources lack computing capabilities but have abundant storage capacity, primarily providing data read and write services to other resource entities. For example, CPUs, GPUs, and TPUs belong to the Master category, while DRAM, NVM, and SSDs belong to the Slave category.
[0044] The interconnect controller is the core component of the entire decoupled resource pool, and its function is to provide a unified network communication interface for different resource entities. Specifically, it includes the following four functions: 1. Data migration function. Different resource entities often need to use different protocols for data exchange. For example, CPUs use the QPI protocol for message passing, GPUs use the PCIe protocol for message passing, and SSDs use the NVMe protocol for data exchange. The data migration function provides a unified abstract access interface for different Masters. The Master only needs to specify <resource type, source address, destination address, data length, request type> to access all resources in the resource pool; 2. Data reduction function. In the resource decoupling architecture, the interconnect network module bears the majority of communication traffic. Therefore, the interconnect controller should provide data reduction functions to reduce communication traffic in the network. For example, data compression and decompression functions; 3. Data scheduling function. Since different resource entities may be accessed by multiple resource entities simultaneously, and these entities have different network communication requirements, the interconnect controller needs to provide data scheduling functionality to differentiate and treat network traffic between different resource entities, scheduling data flows according to a specific scheduling strategy. 4. Network protocol processing functionality. The interconnect controller needs to provide end-to-end data transmission services; therefore, it must rely on a specific network communication protocol. This protocol can be TCP / IP or InfiniBand. This invention does not specify the specific implementation method of the interconnect controller; any hardware unit that meets the above four functions can be used as an interconnect controller.
[0045] An interconnect network is used to interconnect different interconnect controllers. Internally, it can contain multiple switches, multiple routers, or a combination of both. Different strategies can be selected depending on the specific networking requirements.
[0046] The management module manages all resources connected to the interconnected network, including resource detection, address allocation, and address resolution. Resource detection detects all available resources connected to the interconnected network; the address allocation module assigns address information to all available resources; and address resolution provides address lookup services, such as determining the location of a resource within the interconnected network based on its address. The management module records information about all resource entities connected to the interconnected network and maintains a mapping table from resource address to network address for each type of resource entity; each type of resource has an independent address mapping table. The management module can be a server node or a dedicated hardware module with network communication capabilities.
[0047] In some preferred embodiments, the format of the broadcast packet may include packet type, resource type, and resource capacity. The management module uses the following methods for resource detection:
[0048] Send the first broadcast packet, whose packet type is "probe"; all others are empty.
[0049] Receive the second broadcast packet sent by the resource entity. The packet type of the second broadcast packet is "response", "declare" or "remove". The resource type and resource capacity are filled in with the corresponding resource entity information.
[0050] The resource type and resource capacity of the corresponding resource entity are determined based on the received second broadcast packet.
[0051] The management module can dynamically maintain the address mapping table based on the resource detection results.
[0052] If a second broadcast packet of type "response" or "declaration" is received, then according to the detected resource type and resource capacity, a global address to network address mapping is assigned to the corresponding resource entity, and the obtained mapping relationship is added to the address mapping table of the corresponding resource type;
[0053] If a second broadcast packet of type "remove" is received during resource probing, the mapping relationship of the corresponding resource entity is deleted from the address mapping table of the corresponding resource type.
[0054] The address mapping table includes entries for the start address, end address, and network address.
[0055] The following methods can be used to add entries to the address mapping table:
[0056] Use the address mapping table of the resource type corresponding to the detected resource entity as the target address mapping table;
[0057] Fill the starting address of the target address mapping table with the current starting address, calculate the sum of the current starting address of the target address mapping table and the resource capacity and fill it with the ending address, fill the network address of the resource entity with the network address, generate a new table entry and add it to the target address mapping table;
[0058] Update the starting address of the target address mapping table to the address following the calculated ending address.
[0059] The following steps are used to delete entries in the address mapping table:
[0060] Using the network address as an index, search for and remove the entry corresponding to the detected resource entity in the target address mapping table.
[0061] More specifically, the resource exploration method includes the following steps:
[0062] 1. When the management module connects to the interconnected network, it sends broadcast packets to the network. The format of the broadcast packet is <source network address, destination network address, packet type, resource type, capacity>. The packet type is "probe", the resource type is empty, the capacity is empty, and the source and destination network addresses are empty.
[0063] 2. Upon receiving a "probe" type broadcast packet, the resource entity replies to the management module with its resource type and capacity in the form of a broadcast packet response. The format of the broadcast packet response is consistent with that in step 1. The packet type is "response," and the resource type and capacity are filled with the corresponding information for the current resource.
[0064] 3. After receiving a broadcast packet of type "response", the management module determines the corresponding resource entity type and capacity based on the content of the response packet, assigns a global address to network address mapping for the resource, and saves the mapping relationship to the address mapping table of this type of resource;
[0065] 4. When a new resource entity joins the network, it proactively sends a broadcast packet to the network, carrying its resource type and capacity. The packet type is "declaration," and the resource type and capacity are filled with the corresponding information for that resource.
[0066] 5. The management module claims the broadcast packet, determines the corresponding resource entity type and capacity based on the content of the broadcast packet, assigns a global address to network address mapping for the resource, and adds the mapping relationship to the address mapping table of this type of resource;
[0067] 6. When a resource entity needs to be removed from the network, it sends a broadcast packet to the network before being removed. The broadcast packet carries its resource type and resource capacity. The packet type is "Remove," and the resource type and capacity are filled with the corresponding information for that resource.
[0068] 7. The management module claims the broadcast packet, determines the corresponding resource entity type and capacity based on the content of the broadcast packet, and deletes the mapping relationship in the address mapping table.
[0069] The method for mapping resource addresses includes the following steps:
[0070] 1. The management module maintains a separate address mapping table for each resource type. Each address mapping table entry contains: <start address, end address, network address>. During initialization, all address mapping table entries are empty. Each address mapping table maintains a base address; during initialization, the base address of all address mapping tables is 0. Proceed to step 2.
[0071] 2. Perform resource probing and wait for broadcast packets to arrive. If the broadcast packet type is "response" or "declaration," proceed to step 3; if the broadcast packet type is "remove," proceed to step 4.
[0072] 3. The management module detected the addition of a resource entity and confirmed that its resource entity type is A and its capacity is R. cap And the network address where the resource is located is R net If the base address of the address mapping table to which the resource entity belongs is R base Then add an entry to the address mapping table. <R base R base +R cap R net At the same time, the base address R base Updated to (R) base +R cap +1); Proceed to step 2 to continue probing resource entities;
[0073] 4. The management module detected the removal of a resource entity and confirmed that its resource entity type was A and its capacity was R. cap And the network address where the resource is located is R net The management module uses R. net As an index, look up network address R in the address mapping table of Class A resources. net Remove the entry from the table.
[0074] In some preferred embodiments, the management module is also used to coordinate mutual access between different resource entities through command codes with a defined format. A local address mapping table can be stored in the interconnect controller to record the correspondence between local resource address ranges and network addresses, thereby achieving the mapping from local resource addresses to network addresses and avoiding reliance on the remote management module for each address translation. The local mapping table acts as an address cache, reusing historical query results. When the same resource entity accesses the same target address multiple times, only the management module needs to be queried the first time; subsequent requests are directly resolved through the local table, reducing dependence on the management module and network traffic.
[0075] Its resource request method includes the following steps:
[0076] 1. Resource entity A generates a request command with the format <command type, resource type, local resource address, destination resource address, resource length>. The command type can be either a "read command" or a "write command," and the resource type is denoted as R. type The source resource address is denoted as R. s-addr The destination resource address is denoted as R. d-addr The resource length is denoted as R. len ;
[0077] 2. The interconnect controller takes over the command and searches the local address mapping table according to the command. If a match is found, proceed to step 10; otherwise, proceed to step 3.
[0078] 3. The interconnect controller encapsulates the read command into a network data packet, with the packet type being "address query" and the packet content being the read command from step 1; it then sends the data packet to the management module; proceed to step 4;
[0079] 4. The management module receives the "address query" data packet, locates the corresponding address mapping table according to the resource type, obtains the first table entry, and proceeds to step 5 to perform address lookup;
[0080] 5. Retrieve the resource's start address, end address, and network address from the table entry, denoted as R. start R end and R net ; Obtain the resource address R carried in the data packet d-addr If R d-addr Greater than or equal to R start And less than or equal to R end Proceed to step 6; if R d-addr Less than R start Proceed to step 7; if R d-addr Greater than R end And the current entry is the last entry, proceed to step 7; if R d-addr Greater than R end And since the current entry is not the last entry, retrieve the next entry and proceed to step 5;
[0081] 6. If a table entry is found, return the current table entry to the interconnect controller of resource entity A, and proceed to step 8;
[0082] 7. If the table lookup fails, return an error code to the interconnect controller of resource entity A and proceed to step 9;
[0083] 8. The interconnect controller of resource entity A writes the received entries into its local mapping table; proceed to step 10;
[0084] 9. The interconnect controller of resource entity A receives the error code, returns the error code to the resource entity, and terminates the operation;
[0085] 10. The interconnect controller determines and generates a resource request packet. The packet format is <request type, network address, starting address, length>. The request type is the command type from step 1, and the network address is R... net The starting address is (R d-addr –R start ), with a length of R len If the command type is "read command", proceed to step 11; if the command type is "write command", proceed to step 12.
[0086] 11. Send data packets directly to the interconnected network;
[0087] 12. Interconnection controller according to R s-addr and R len Data is read from resource entity A, encapsulated in a network packet, and sent to the interconnected network.
[0088] Its resource response method includes the following steps:
[0089] 1. The interconnect controller of resource entity B receives a resource request packet, parses the packet, and obtains the request type and source resource address R. s-addr Target resource address R d-addr Information and length information R len If the request type is "write request", proceed to step 2; if the request type is "read request", proceed to step 3.
[0090] 2. The interconnect controller is based on R d-addr and R len The message writes the data carried in the data packet into the address space of resource entity B, and the response is complete;
[0091] 3. The interconnect controller is based on R d-addr and R len The system reads the corresponding data from the address space of resource entity B, encapsulates the data in a network packet, and sends it to the interconnect controller of resource entity A. The interconnect controller of resource entity A writes the received response data into the address space of resource entity A, and the response is complete.
[0092] A preferred embodiment 1 of this implementation is a specific implementation case of a network-centric resource decoupling system, such as... Figure 2 As shown, it includes: a GPU resource pool, an SSD resource pool, an NPU resource pool, a management module, and an interconnect network. The interconnect network and interconnect controller support the InfiniBand network protocol.
[0093] GPU resource pool: A resource pool consisting of multiple GPUs and an interconnect controller. All GPUs and the interconnect controller are interconnected via a PCIe bus;
[0094] SSD resource pool: A resource pool consisting of multiple SSDs and an interconnect controller. All SSDs and the interconnect controller are interconnected via a PCIe bus;
[0095] NPU Resource Pool: A resource pool consisting of multiple NPUs and one interconnect controller. All NPUs and the interconnect controller are interconnected via a PCIe bus;
[0096] The management module is an independent host node that runs a management program capable of processing network data packets;
[0097] The interconnect network is an InfiniBand switch, through which management modules and interconnect controllers for different resource pools are connected.
[0098] A preferred embodiment 2 of this implementation is a detection process, including the following steps:
[0099] 1. After the management module is connected to the switch, it sends broadcast packets to the network;
[0100] 2. When the SSD resource pool receives a "probe" type broadcast packet, it replies to the management module with the SSD resource type and resource capacity in the form of a broadcast packet response;
[0101] 3. When the NPU resource pool receives a "probe" type broadcast packet, it replies to the management module with the NPU's resource type and resource capacity in the form of a broadcast packet response;
[0102] 4. When the GPU resource pool receives a "probe" type broadcast packet, it replies to the management module with the GPU resource type and resource capacity in the form of a broadcast packet response;
[0103] 5. After receiving a broadcast packet of type "response", the management module determines the resource type to which the broadcast packet belongs. If it is an SSD resource, proceed to step 6; if it is an NPU resource, proceed to step 7; if it is a GPU resource, proceed to step 8.
[0104] 6. Allocate address mapping entries for SSD resources;
[0105] 7. Allocate address mapping entries for NPU resources;
[0106] 8. Allocate address mapping entries for GPU resources.
[0107] A preferred embodiment 3 of this implementation is a global address to network address mapping process, including the following steps:
[0108] 1. The management module receives a broadcast packet of type "response" and determines that the resource type of the broadcast packet is SSD; proceed to step 2 to allocate address mapping table entries;
[0109] 2. Process the address mapping table of the SSD resource entity, obtain the current base address, assuming it is 0x10000; proceed to step 3;
[0110] 3. The SSD resource entity's capacity is 0xFFFF and its network address is 10.18.20.95, obtained from the response packet. Proceed to step 4 to add a table entry;
[0111] 4. Generate an entry <0x10000, 0x1FFFF, 10.18.20.95>, insert this entry into the address mapping table of the SSD resource entity, and update its entry base address to 0x20000.
[0112] The second embodiment of the present invention relates to a resource request and response method, which is applied to the system described above, such as... Figure 3 As shown, it includes the following steps:
[0113] The first resource entity issues a resource request command. The command format includes command type, resource type, local resource address, destination resource address, and resource length. The command type includes "write request" and "read request".
[0114] In response to a resource request command, the interconnect controller of the first resource entity looks up the mapping from the destination resource address to the network address in the local address mapping table. If a match is found, it interacts with the interconnect controller of the second resource entity based on the command type. If no match is found, it encapsulates the resource request command into a first data packet and sends it.
[0115] The management module parses the first data packet received, locates the target address mapping table according to the parsed resource type, and looks up the network address of the target resource. If no match is found, the interconnect controller returns an error code to the first resource entity and terminates the resource request and response process. If a match is found, the matched entry is returned to the interconnect controller of the first resource entity.
[0116] The interconnect controller of the first resource entity writes the entries returned by the management module into the local address mapping table and interacts with the interconnect controller of the second resource entity according to the command type.
[0117] More specifically, if the command type is "write request":
[0118] The interconnection controller of the first resource entity reads the target write data according to the local resource address and resource length, and encapsulates it together with the command type, source network address, destination network address, destination start address and resource length into a second data packet for transmission.
[0119] The interconnection controller of the second resource entity parses the received second data packet, extracts the target write data, and writes it into the corresponding address space of the second resource entity.
[0120] If the command type is "read request":
[0121] The interconnection controller of the first resource entity encapsulates the command type, network address, starting address, and resource length into a second data packet for transmission.
[0122] The interconnect controller of the second resource entity parses the received second data packet, reads the target read data according to the local resource address and resource length, and encapsulates it into a third data packet and sends it to the interconnect controller of the first resource entity.
[0123] The interconnect controller of the first resource entity extracts the target read data from the third data packet and writes it to the resource entity.
[0124] In some preferred embodiments, the resource request method includes the following steps:
[0125] 1. Resource entity A generates a request command with the format <command type, resource type, local resource address, destination resource address, resource length>. The command type can be either a "read command" or a "write command," and the resource type is denoted as R. type The source resource address is denoted as R. s-addr The destination resource address is denoted as R. d-addr The resource length is denoted as R. len ;
[0126] 2. The interconnect controller takes over the command and searches the local address mapping table according to the command. If a match is found, proceed to step 4; otherwise, proceed to step 3.
[0127] 3. The interconnect controller encapsulates the read command into a network data packet, with the packet type being "address query" and the packet content being the read command from step 1; it then sends the data packet to the management module; proceed to step 4;
[0128] 4. The management module receives the "address query" data packet, locates the corresponding address mapping table according to the resource type, obtains the first table entry, and proceeds to step 5 to perform address lookup;
[0129] 5. Retrieve the resource's start address, end address, and network address from the table entry, denoted as R. start R end and R net ; Obtain the resource address R carried in the data packet d-addr If R d-addr Greater than or equal to R start And less than or equal to R end Proceed to step 6; if R d-addr Less than R start Proceed to step 7; if R d-addr Greater than R end And the current entry is the last entry, proceed to step 7; if R d-addr Greater than R end And since the current entry is not the last entry, retrieve the next entry and proceed to step 5;
[0130] 6. If a table entry is found, return the current table entry to the interconnect controller of resource entity A, and proceed to step 8;
[0131] 7. If the table lookup fails, return an error code to the interconnect controller of resource entity A and proceed to step 9;
[0132] 8. The interconnect controller of resource entity A writes the received entries into its local mapping table; proceed to step 10;
[0133] 9. The interconnect controller of resource entity A receives the error code, returns the error code to the resource entity, and terminates the operation;
[0134] 10. The interconnect controller determines and generates a resource request packet. The packet format is <request type, network address, starting address, length>. The request type is the command type from step 1, and the network address is R... net The starting address is (R d-addr –R start ), with a length of R len If the command type is "read command", proceed to step 11; if the command type is "write command", proceed to step 12.
[0135] 11. Send data packets directly to the interconnected network;
[0136] 12. Interconnection controller according to R s-addr and R len Data is read from resource entity A, encapsulated in a network packet, and sent to the interconnected network.
[0137] The resource response method includes the following steps:
[0138] 1. The interconnect controller of resource entity B receives a resource request packet, parses the packet, and obtains the request type and source resource address R. s-addr Target resource address R d-addr Information and length information R len If the request type is "write request", proceed to step 2; if the request type is "read request", proceed to step 3.
[0139] 2. The interconnect controller is based on R d-addr and R len The message writes the data carried in the data packet into the address space of resource entity B, and the response is complete;
[0140] 3. The interconnect controller is based on R d-addr and R len The system reads the corresponding data from the address space of resource entity B, encapsulates the data in a network packet, and sends it to the interconnect controller of resource entity A. The interconnect controller of resource entity A writes the received response data into the address space of resource entity A, and the response is complete.
[0141] A preferred embodiment 4 of this implementation is a resource request and response process, including the following steps:
[0142] 1. The GPU generates a request command with the format <command type, resource type, local resource address, destination resource address, resource length>. The command type is "read command", the resource type is SSD, the source resource address is 0x1728, the destination resource address is 0x9284, and the resource length is 512KB.
[0143] 2. The interconnect controller takes over the command and searches the local address mapping table according to the command. If a match is found, proceed to step 10; otherwise, proceed to step 3.
[0144] 3. The interconnect controller encapsulates the read command into a network data packet, with the packet type being "address query" and the packet content being the read command from step 1; it then sends the data packet to the management module; proceed to step 4;
[0145] 4. The management module receives the "address query" data packet, locates the corresponding address mapping table according to the resource type, obtains the first table entry, and proceeds to step 5 to perform address lookup;
[0146] 5. Obtain the resource entry with a start address of 0x9000, an end address of 0x9FFF, and a network address of 10.18.94.15; obtain the destination resource address 0x9284 carried in the data packet. Comparison shows that 0x9284 is between 0x9000 and 0x9FFFF, proceed to step 6;
[0147] 6. If a table entry is found, return the current table entry to the interconnect controller of the resource entity GPU, and proceed to step 7;
[0148] 7. The interconnect controller of the GPU resource pool writes the received entries into its own local mapping table; proceed to step 9;
[0149] 9. The interconnect controller generates a resource request packet with the format <request type, source network address, destination network address, source start address, destination start address, length>. The request type is "read request", the source network address is 10.18.94.26, the destination network address is 10.18.94.15, the start address is 0x9284, and the length is 512KB. This packet is then sent to the interconnect network.
[0150] 10. The InfiniBand switch forwards packets to the interconnect controller of the SSD resource pool based on the destination network address;
[0151] 11. The interconnect controller of the SSD resource pool receives a resource request packet, parses the packet, and obtains that the packet type is a read request, the source network address is 10.18.94.26, the source start address is 0x1728, the destination start address is 0x9284, and the length is 512KB.
[0152] 12. The interconnect controller reads the corresponding data from the address space of the resource entity SSD via the PCIe bus according to the destination start address and length information, and encapsulates the data in a network packet and sends it to the interconnect controller of resource entity A. The network packet type is "read response", the destination network address is 10.18.94.26, and the destination start address is 0x1728;
[0153] 13. The interconnect controller of the resource entity GPU receives a read response packet with a destination start address of 0x1728 and a length of 512KB. The interconnect controller writes the received response data into the address space of the resource entity GPU through the PCIe bus, and the response is complete.
Claims
1. A resource decoupling system applied to a data center, characterized in that, The system comprises a plurality of resource pools of different resource types and a management module, each resource pool comprises a plurality of resource entities and an interconnection controller providing data communication service for the resource entities; the management module is in communication connection with each interconnection controller, and is configured to perform resource detection of each resource entity by using a broadcast packet of a set format, create an independent address mapping table for each resource type and dynamically maintain the address mapping table according to the resource detection result, and coordinate mutual access among different resource entities by using a command of a set format; the interconnection controller of each resource entity caches the address mapping table corresponding to the resource accessed by the resource entity locally, so as to accelerate the address lookup process in subsequent resource request.
2. The system of claim 1, wherein, The format of the broadcast packet comprises a packet type, a resource type and a resource capacity.
3. The system of claim 2, wherein, The management module performs resource detection by using the following method: a first broadcast packet is sent, the packet type of the first broadcast packet is "detection", and the others are empty; a second broadcast packet sent by a resource entity is received, the packet type of the second broadcast packet is "response", "declaration" or "removal", and the resource type and the resource capacity of the second broadcast packet are filled with corresponding resource entity information; the resource type and the resource capacity of the corresponding resource entity are determined according to the received second broadcast packet.
4. The system of claim 3, wherein, The dynamic maintenance according to the resource detection result comprises: if the second broadcast packet with the packet type of "response" or "declaration" is received, a global address to network address mapping is allocated to the corresponding resource entity according to the detected resource type and the resource capacity, and the obtained mapping relationship is added to the address mapping table of the corresponding resource type; if the second broadcast packet with the packet type of "removal" is received during resource detection, the mapping relationship of the corresponding resource entity is deleted from the address mapping table of the corresponding resource type.
5. The system of claim 4, wherein, The table entry of the address mapping table comprises a start address, an end address and a network address.
6. The system of claim 5, wherein, The adding of the obtained mapping relationship to the address mapping table of the corresponding resource type comprises: the address mapping table of the detected resource entity corresponding to the resource type is taken as a target address mapping table; a new table entry is generated and added to the target address mapping table by filling the current start address of the target address mapping table into the start address, calculating the sum of the current start address of the target address mapping table and the resource capacity and filling the sum into the end address, and filling the network address of the resource entity into the network address; the start address of the target address mapping table is updated to the next address of the calculated end address.
7. The system of claim 6, wherein, The deleting of the mapping relationship of the corresponding resource entity from the address mapping table of the corresponding resource type comprises: the table entry corresponding to the detected resource entity is found and removed from the target address mapping table by taking the network address as an index.
8. A resource request and response method, characterized by, The method is applied to the system of claims 1-7, and comprises the following steps: a resource request command is sent by a first resource entity, the format of the command comprises a command type, a resource type, a local resource address, a destination resource address and a resource length, and the command type comprises "write request" and "read request"; The interconnection controller of the first resource entity looks up the mapping of the destination resource address to the network address in the local address mapping table in response to the resource request command, and if a hit is obtained, carries out data interaction with the interconnection controller of the second resource entity according to the command type, and if no hit is obtained, encapsulates the resource request command into a first data packet and sends the first data packet; The management module parses the received first data packet, locates a target address mapping table according to the parsed resource type, and looks up the network address of the destination resource, if no hit is obtained, returns an error code to the first resource entity via the interconnection controller, and terminates the resource request and response process, and if a hit is obtained, returns the hit table item to the interconnection controller of the first resource entity; The interconnection controller of the first resource entity writes the table item returned by the management module into the local address mapping table, and carries out data interaction with the interconnection controller of the second resource entity according to the command type.
9. The method of claim 8, wherein, The data interaction with the interconnection controller of the second resource entity according to the command type comprises: If the command type is "write request": The interconnection controller of the first resource entity reads target write data according to the local resource address and the resource length, encapsulates the command type, the source network address, the destination network address, the destination start address and the resource length into a second data packet and sends the second data packet; The interconnection controller of the second resource entity parses the received second data packet, extracts the target write data and writes the target write data into the corresponding address space of the second resource entity; If the command type is "read request": The interconnection controller of the first resource entity encapsulates the command type, the network address, the start address and the resource length into a second data packet and sends the second data packet; The interconnection controller of the second resource entity parses the received second data packet, reads target read data according to the local resource address and the resource length, and encapsulates the target read data into a third data packet and sends the third data packet to the interconnection controller of the first resource entity; The interconnection controller of the first resource entity extracts the target read data from the third data packet and writes the target read data into the resource entity.