Dual control memory interconnect system and transmission control method

Abstract

The application is suitable for the technical field of data storage, and provides a double-control storage interconnection system and a transmission control method.The system comprises a main controller, a backup controller, a first NTB channel, a second NTB channel and a hardware monitoring unit.The main controller and the backup controller each comprise a CPU and a PCIe Switch chip.The hardware monitoring unit is used for acquiring connection state information of the first NTB channel and the second NTB channel, and sending the connection state information to the first CPU and / or the second CPU.When the communication states of the first NTB channel and the second NTB channel are both normal, the first NTB channel is used for transmitting the connection state information and heartbeat detection information.The second NTB channel is used for transmitting to-be-synchronized cache data and cross-controller IO data.The application has the advantages that, compared with the prior art, the risk of heartbeat delay and state synchronization delay caused by large data transmission is significantly reduced, and the stability of the double-control storage system in the processes of cache synchronization, heartbeat detection and cross-controller IO transmission is improved.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, and in particular relates to a dual-controller storage interconnection system and transmission control method. Background Technology

[0002] Dual-controller storage systems typically employ two storage controllers, allowing the other to continue handling storage operations should one controller fail. The two controllers need to perform tasks such as cache mirroring, heartbeat detection, fault communication, transaction log synchronization, and cross-controller input / output transfers.

[0003] In existing dual-controller storage systems, high-speed communication between controllers can be achieved using non-transparent bridges (NTBs). Common methods fall into two categories: one utilizes the processor's built-in non-transparent bridge engine to establish the interconnection between controllers; the other establishes the interconnection through the non-transparent bridge port integrated into an external PCIe switching chip. The former offers lower latency and is suitable for transmitting short control messages such as heartbeats, fault commands, and register interactions; the latter offers higher bandwidth and is suitable for transmitting service data such as cache mirroring, cross-controller I / O, and large blocks of direct memory access.

[0004] However, existing solutions typically use a single NTB to carry all interactions between the two controllers. If the processor's PCIe root port, PCIe switch chip, backplane high-speed differential connection, or corresponding non-transparent bridge port malfunctions, high-speed communication between the two controllers is easily interrupted. Cache mirroring and fault interaction may be affected, and the storage system may degrade from a dual-controller operating state to a single-controller operating state, impacting the continuous operation of core storage services. Simultaneously, in a single non-transparent bridge, control messages such as heartbeat messages, fault commands, and register configuration commands are often mixed with high-bandwidth service data such as cache dirty data mirroring, cross-controller I / O, and large block direct memory access synchronization data. When the service data transmission volume is large, control messages may experience bandwidth congestion, leading to heartbeat detection delays and fault command delivery lags, thereby causing misjudgments, incorrect switching, or data inconsistency risks.

[0005] Therefore, existing dual-controller storage interconnection solutions suffer from high failure risk and competition between control traffic and data traffic, making it difficult to meet the comprehensive requirements of enterprise-level NVMe dual-controller storage systems for high reliability, low latency control interaction, and high bandwidth data synchronization. Summary of the Invention

[0006] The purpose of this application is to provide a dual-controller storage interconnect system, which aims to solve the problems of high failure risk, mutual contention between control traffic and data traffic in existing dual-controller storage interconnect solutions, making it difficult to meet the comprehensive requirements of enterprise-level NVMe dual-controller storage systems for high reliability, low latency control interaction and high bandwidth data synchronization.

[0007] This application provides a dual-controller storage interconnect system, characterized in that the system includes:

[0008] Main controller, backup controller, first NTB channel, second NTB channel, and hardware monitoring unit;

[0009] The main controller includes a first CPU and a first PCIe Switch chip;

[0010] The backup controller includes a second CPU and a second PCIe switch chip;

[0011] The first NTB channel includes a first NTB functional unit disposed in the first CPU, a second NTB functional unit disposed in the second CPU, and a first PCIe line group connecting the first NTB functional unit and the second NTB functional unit.

[0012] The second NTB channel includes a first NTB port disposed in the first PCIe Switch chip, a second NTB port disposed in the second PCIe Switch chip, and a second PCIe line group connecting the first NTB port and the second NTB port;

[0013] The first PCIe line group and the second PCIe line group are independent of each other;

[0014] The hardware monitoring unit is used to acquire the connection status information of the first NTB channel and the second NTB channel, and send the connection status information to the first CPU and / or the second CPU.

[0015] When both the first NTB channel and the second NTB channel are in normal connectivity, the first NTB channel is used to transmit connection status information and heartbeat detection information; the second NTB channel is used to transmit cached data to be synchronized and cross-controller I / O data.

[0016] Preferably, the hardware monitoring unit includes a first CPLD chip disposed in the main controller and a second CPLD chip disposed in the backup controller;

[0017] The first CPLD chip is connected to the first retimer chip and the first PCIeSwitch chip in the main controller, and the first retimer chip is connected to the first CPU.

[0018] The second CPLD chip is connected to the second retimer chip and the second PCIeSwitch chip in the backup controller, and the second retimer chip is connected to the second CPU;

[0019] The first CPLD chip is used to obtain the connection status of the first NTB channel and the second NTB channel on the main controller side, and the second CPLD chip is used to obtain the connection status of the first NTB channel and the second NTB channel on the backup controller side.

[0020] Preferably, the first CPLD chip sends a status read command word through the I2C channel to obtain the connection status of the first NTB channel and the second NTB channel on the main controller side, and obtains the connection status of the first NTB channel and the second NTB channel on the backup controller side through the GPIO channel.

[0021] The second CPLD chip sends a status read command word through the I2C channel to obtain the connection status of the first NTB channel and the second NTB channel on the backup controller side, and obtains the connection status of the first NTB channel and the second NTB channel on the main controller side through the GPIO channel.

[0022] Preferably, the first NTB channel and the second NTB channel are each configured with an independent address mapping table, BAR space and DMA channel;

[0023] The remote memory mapping region corresponding to the first NTB channel is physically isolated from the remote memory mapping region corresponding to the second NTB channel.

[0024] Preferably, the connection status information includes connectivity status, bit error rate, and negotiation rate;

[0025] The connectivity status indicates whether the first NTB channel and the second NTB channel are disconnected, the bit error status indicates whether bit errors occur in the first NTB channel and the second NTB channel, and the negotiation rate indicates the data transmission rate of the first NTB channel and the second NTB channel.

[0026] Preferably, the first CPU and / or the second CPU are used to control the transmission channels of the cached data to be synchronized, the cross-controller I / O data, the connection status information, and the heartbeat detection information based on the connection status information.

[0027] Preferably, the method by which the first CPU and / or the second CPU controls the data transmission channel based on connection status information includes:

[0028] When the first NTB channel is detected to be faulty and the second NTB channel is normal, the first CPU and the second CPU transmit the connection status information and the heartbeat detection information through the second NTB channel.

[0029] When both the first NTB channel and the second NTB channel fail, the first CPLD chip and the second CPLD chip synchronize the NTB connection status of both parties and notify the first CPU and the second CPU respectively to perform fault isolation processing.

[0030] The fault isolation process includes stopping the writing of new cached data to the cache area to be synchronized.

[0031] Preferably, the method for controlling the data transmission channel based on connection status information further includes:

[0032] Once the connection between the first NTB channel and the second NTB channel is restored to normal, the first CPU and the second CPU perform incremental data synchronization based on the transaction log, and enable the first NTB channel to resume transmitting the connection status information and the heartbeat detection information, and enable the second NTB channel to resume transmitting the cached data to be synchronized and the cross-controller IO data.

[0033] Preferably, when the second NTB channel experiences bit errors, disconnections, or negotiation rates below the minimum rate threshold, and the first NTB channel does not experience bit errors, disconnections, or negotiation rates below the minimum rate threshold, the first CPU and the second CPU transmit the cached data to be synchronized through the first NTB channel.

[0034] Another objective of this application is to provide a transmission control method for a dual-controller memory interconnect, the method being applied to the dual-controller memory interconnect system described above;

[0035] The method includes:

[0036] Obtain connection status information input from the hardware monitoring unit;

[0037] When neither the first NTB channel nor the second NTB channel experiences bit errors, disconnections, or negotiation rates below the minimum rate threshold, the connection status information and the heartbeat detection information are transmitted through the first NTB channel, and the cached data to be synchronized and the cross-controller I / O data are transmitted through the second NTB channel.

[0038] When the second NTB channel experiences bit errors, disconnections, or negotiation rates below the minimum rate threshold, and the first NTB channel does not experience bit errors, disconnections, or negotiation rates below the minimum rate threshold, the cached data to be synchronized is transmitted through the first NTB channel.

[0039] When the first NTB channel experiences a bit error, disconnection, or negotiation rate below the minimum rate threshold, and the second NTB channel does not experience a bit error, disconnection, or negotiation rate below the minimum rate threshold, the connection status information and the heartbeat detection information are transmitted through the second NTB channel.

[0040] When both the first NTB channel and the second NTB channel are disconnected, stop writing new cached data to the buffer to be synchronized.

[0041] Once the connection between the first NTB channel and the second NTB channel is restored to normal, incremental synchronization of data between the primary controller and the backup controller is performed based on the transaction log. The connection status information and heartbeat detection information transmitted by the first NTB channel and the cached data to be synchronized and the cross-controller IO data transmitted by the second NTB channel are restored.

[0042] This application provides a dual-controller storage interconnection system, the key advantage of which is that it reduces the risk of a single NTB hardware failure causing a complete interruption of cross-controller communication between the primary and backup controllers. By utilizing the different characteristics of the two types of NTB hardware, it avoids the problem that a single hardware path cannot simultaneously handle low-latency control and high-bandwidth data transmission. The CPU-side NTB channel is close to the CPU, suitable for transmitting heartbeat detection information and connection status information; the PCIe Switch-side NTB channel, combined with the high-bandwidth forwarding capability of the PCIe switching chip, is suitable for transmitting cached data to be synchronized and cross-controller I / O data; the hardware monitoring unit enables the status of the two NTB channels to be independently acquired and provided to the CPU, allowing the CPU to determine whether the connection is normal based on the channel status, thus improving the stability of the dual-controller storage system during cache synchronization, heartbeat detection, and cross-controller I / O transmission. Attached Figure Description

[0043] Figure 1 A structural block diagram of a dual-controller storage interconnect system provided in this application embodiment;

[0044] Figure 2 A flowchart illustrating the steps of a transmission control method for dual-controller storage interconnection provided in this application embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish the first unit or module from another unit or module. For example, without departing from the scope of this application, the first unit may be referred to as the second unit, and similarly, the second module may be referred to as the first module.

[0047] like Figure 1 As shown, in one embodiment, a dual-controller storage interconnect system is proposed, the system comprising:

[0048] The system comprises a main controller, a backup controller, a first NTB channel, a second NTB channel, and a hardware monitoring unit. The main controller includes a first CPU and a first PCIe Switch chip. The backup controller includes a second CPU and a second PCIe Switch chip. The first NTB channel includes a first NTB functional unit within the first CPU, a second NTB functional unit within the second CPU, and a first PCIe line group connecting the first NTB functional unit and the second NTB functional unit. The second NTB channel includes a first NTB port within the first PCIe Switch chip, a second NTB port within the second PCIe Switch chip, and a second PCIe line group connecting the first NTB port and the second NTB port. The first PCIe line group and the second PCIe line group are independent of each other. The hardware monitoring unit is used to acquire the connection status information of the first NTB channel and the second NTB channel and send the connection status information to the first CPU and / or the second CPU. When the connection status of both the first NTB channel and the second NTB channel is normal, the first NTB channel is used to transmit connection status information and heartbeat detection information. The second NTB channel is used to transmit cached data to be synchronized and cross-controller I / O data.

[0049] In the embodiments of this application, such as Figure 1 As shown, NTB stands for Non-Transparent Bridge, used to establish a controlled communication connection between two independent PCIe host domains. The PCIe Switch chip is a PCIe switching chip used to expand and forward PCIe channels within the controller. The first PCIe Switch chip is located in the primary controller, and the second PCIe Switch chip is located in the backup controller. Besides providing PCIe port expansion functionality, both also provide NTB ports for cross-controller communication. A "normal connectivity" state indicates that both channels have experienced disconnections, bit errors, and the negotiated rate meets transmission requirements.

[0050] Those skilled in the art will understand that the NTB functional unit within the CPU is a non-transparent bridge functional module provided by the CPU, used to establish a controlled communication relationship between two independent CPU host domains. In the embodiments of this application, the first CPU and the second CPU each have independent PCIe address spaces and are not directly enumerated by the other like ordinary PCIe devices; the first NTB functional unit and the second NTB functional unit are used to establish an isolated control communication channel between the two CPUs. The first PCIe line group is a high-speed differential PCIe connection connecting the first NTB functional unit and the second NTB functional unit, and can be set on an interconnect backplane, connection board, cable, or other board-level interconnect structure. The second PCIe line group is a high-speed differential PCIe connection connecting the first NTB port and the second NTB port. It can be implemented using a backplane connection, that is, through PCIe differential traces and their connector contact paths set on the interconnect backplane. The first PCIe line group and the second PCIe line group are independent of each other and do not share the same set of high-speed PCIe connections. Connection status information refers to information indicating the connection status, error rate, and negotiation rate of the first NTB channel and the second NTB channel. Heartbeat detection information is a liveness confirmation message periodically exchanged between the first CPU and the second CPU, used to determine whether the peer controller is still operational. Data to be synchronized in the cache refers to data that has been written to the local controller's cache but has not yet been synchronized to the peer controller's cache. Cross-controller I / O data refers to the I / O requests, I / O responses, and corresponding read / write data generated when one controller needs to complete read / write access through the other controller. The reason for this division of labor is that connection status and heartbeat information data volumes are small but latency-sensitive, making them suitable for the CPU-side NTB channel; cache synchronization and cross-controller I / O data volumes are large, making them suitable for the PCIeSwitch-side NTB channel. This avoids large-capacity data transmission crowding out the heartbeat detection and status synchronization channels. The primary and backup controllers each have independent CPU, PCIe address spaces, and device enumeration spaces. The first NTB channel allows the first CPU and the second CPU to exchange control information without requiring one CPU to directly enumerate all PCIe devices in the other controller. The second NTB channel allows the first PCIe Switch chip and the second PCIe Switch chip to exchange cross-controller data; its address mapping space is configured separately from the first NTB channel.

[0051] In this embodiment, the first NTB channel and the second NTB channel are configured to handle different data types. The first NTB channel primarily handles control information that is latency-sensitive, has a small data volume, and needs to be continuously present, such as connection status information and heartbeat detection information. The second NTB channel primarily handles cross-controller data with a larger data volume, such as data to be synchronized and cross-controller I / O data. This configuration is because heartbeat detection information and connection status information cannot be blocked by large-capacity cached synchronization data; while data to be synchronized and cross-controller I / O data require higher bandwidth. After the two types of information are transmitted through separate channels, control information will not compete with large-capacity data for the same NTB channel resources. Existing mixed transmission methods are prone to causing heartbeat timeouts and delays in fault commands.

[0052] In this application, cached data awaiting synchronization refers to cached data that has been written to the local controller's cache but has not yet been synchronized to the peer controller. For example, after the primary controller receives a write request from the host, the data first enters the primary controller's cache. At this time, this data needs to be synchronized to the backup controller so that the backup controller can continue to provide consistent data when the primary controller fails. Cross-controller I / O data refers to I / O data that requires cooperation from another controller to complete, including cross-controller read / write requests and the data corresponding to those requests.

[0053] Under normal conditions, the first NTB channel continuously transmits connection status information and heartbeat detection information, while the second NTB channel transmits cached data to be synchronized and cross-controller I / O data. For example, when the host writes a data block to the main controller, the main controller sends the corresponding cached data to be synchronized to the backup controller via the second NTB channel; simultaneously, the first CPU and the second CPU exchange heartbeat detection information and connection status information via the first NTB channel. This process allows data synchronization and control monitoring to occur simultaneously, without them occupying the same transmission channel.

[0054] In existing dual-controller storage systems, heartbeat detection information, connection status information, cache synchronization data, and cross-controller I / O data typically share the same NTB channel. When the amount of cache data to be synchronized or cross-controller I / O data is large, it consumes NTB channel bandwidth, preventing timely transmission of heartbeat detection and connection status information. Delayed heartbeat information may lead the controller to misjudge the peer as abnormal; delayed connection status information makes it difficult for the system to detect channel abnormalities in a timely manner. This application fixes the transmission of connection status information and heartbeat detection information through the first NTB channel, and fixes the transmission of cache data to be synchronized and cross-controller I / O data through the second NTB channel. This prevents control information with small data volumes and low latency requirements from competing for the same channel resources with large amounts of cache synchronization data, thereby reducing the risks of heartbeat timeouts, state synchronization delays, and erroneous switching.

[0055] In this application, the first NTB channel relies on the NTB functional unit within the CPU, while the second NTB channel relies on the NTB port within the PCIe Switch chip. The two channels are not logically multiplexed on the same PCIe bus or simply partitioned on the same hardware unit. This ensures that if one channel malfunctions, the other channel still maintains its independent physical and communication foundation. Compared to solutions using only CPU NTB or only PCIe Switch NTB, this application reduces the risk of a single NTB hardware malfunction causing a complete interruption of cross-controller communication between the primary and backup controllers. This application utilizes the different characteristics of the two types of NTB hardware to avoid the problem that a single hardware path cannot simultaneously handle low-latency control and high-bandwidth data transmission. The CPU-side NTB channel, being close to the CPU, is suitable for transmitting heartbeat detection information and connection status information; the PCIe Switch-side NTB channel, leveraging the high-bandwidth forwarding capability of the PCIe switching chip, is suitable for transmitting data to be synchronized and cached, as well as cross-controller I / O data. Existing single-CPU NTB solutions consume CPU resources when synchronizing large amounts of cache, making it difficult to handle continuous large data transfers. While existing single-PCIe SwitchNTB solutions are suitable for large data transfers, control information is easily affected by high-volume business data when simultaneously carrying heartbeat and status information. This application allocates transmission channels according to information type, allowing the CPU-side NTB and Switch-side NTB to each undertake more suitable tasks.

[0056] The hardware monitoring unit in this application enables the status of the two NTB channels to be acquired independently and provided to the CPU. This allows the CPU to determine whether the connection is normal, or whether there are bit errors, or whether the negotiation rate meets the requirements, based on the channel status. This improves the stability of the dual-controller storage system during cache synchronization, heartbeat detection, and cross-controller I / O transmission.

[0057] In a preferred embodiment, the hardware monitoring unit includes a first CPLD chip disposed in the main controller and a second CPLD chip disposed in the backup controller; the first CPLD chip is connected to a first retimer chip and a first PCIe switch chip in the main controller, and the first retimer chip is connected to a first CPU; the second CPLD chip is connected to a second retimer chip and a second PCIe switch chip in the backup controller, and the second retimer chip is connected to a second CPU; the first CPLD chip is used to acquire the connection status of a first NTB channel and a second NTB channel on the main controller side, and the second CPLD chip is used to acquire the connection status of a first NTB channel and a second NTB channel on the backup controller side.

[0058] In this embodiment, the first CPLD chip and the second CPLD chip are hardware status monitoring devices. The CPLD chip reads or receives the PCIe connection status and provides the status to the corresponding CPU. The first retimer chip is located on the PCIe high-speed channel related to the first CPU in the main controller, and is used for retiming, equalizing, and restoring the PCIe high-speed signal; the second retimer chip is located on the PCIe high-speed channel related to the second CPU in the backup controller, and has the same function. The CPLD chip is connected to the retimer chip to obtain the PCIe connection status corresponding to the NTB channel on the CPU side; the CPLD chip is connected to the PCIeSwitch chip to obtain the PCIe connection status corresponding to the NTB channel on the Switch side. The first NTB channel connection status on the main controller side refers to the connection status exhibited by the first CPU, the first NTB functional unit, and the first PCIe line group at the main controller end. The second NTB channel connection status on the main controller side refers to the connection status exhibited by the first PCIe Switch chip, the first NTB port, and the second PCIe line group at the main controller end. Similarly, the connection status of the first NTB channel and the second NTB channel on the backup controller side correspond to each other. The first retimer chip is set on the PCIe connection path related to the first CPU to facilitate high-speed signal transmission of the NTB connection on the first CPU side. The first CPLD chip is connected to the first retimer chip to obtain the PCIe status related to the NTB connection on the first CPU side; the first CPLD chip is connected to the first PCIe switch chip to obtain the PCIe status of the second NTB channel on the main controller side. Figure 1 The retimer chip in the diagram points to the PCIe status, which in turn points to the CPLD chip. This indicates the direction of PCIe connection status acquisition, but does not mean that NTB data is forwarded via the CPLD. In other words, the CPLD chip only reads or receives status data; it does not carry cached data to be synchronized or cross-controller I / O data. This avoids the status monitoring process consuming CPU software polling resources or introducing the CPLD into the high-speed data path.

[0059] In a preferred embodiment, the first CPLD chip sends a status read command word through the I2C channel to obtain the connection status of the first NTB channel and the second NTB channel on the main controller side, and obtains the connection status of the first NTB channel and the second NTB channel on the backup controller side through the GPIO channel; the second CPLD chip sends a status read command word through the I2C channel to obtain the connection status of the first NTB channel and the second NTB channel on the backup controller side, and obtains the connection status of the first NTB channel and the second NTB channel on the main controller side through the GPIO channel.

[0060] In this embodiment, the I2C channel is used by the CPLD chip to read the status of its local device. The first CPLD chip sends a status read command word to the retimer chip or the first PCIe switch chip in the main controller via the I2C channel, thereby obtaining the connection status of the first NTB channel and the second NTB channel on the main controller side. The second CPLD chip obtains the corresponding status on the backup controller side via the I2C channel. The status read command word is used to trigger a hardware command for the read device to return the PCIe connection status. The GPIO channel is used to synchronize status information between the two controllers. The first CPLD chip obtains the connection status of the first NTB channel and the second NTB channel on the backup controller side via the GPIO channel; the second CPLD chip obtains the connection status of the first NTB channel and the second NTB channel on the main controller side via the GPIO channel. By using I2C to read the local status and GPIO to obtain the peer status, both CPLDs can grasp the status of both sides, avoiding single-end misjudgment. For example, if the main controller side detects that the second NTB channel is disconnected, but the backup controller side still shows that the connection is not disconnected, the system can first perform status synchronization, and then the CPU can perform transmission channel adjustment according to the synchronized status. In this embodiment, "local end" refers to the controller side where the CPLD chip is located. The local end of the first CPLD chip is the master controller, and the local end of the second CPLD chip is the backup controller. "Reciprocal end" refers to the other controller side. In this way, both the master controller side and the backup controller side can grasp the status of the NTB channels of both sides. The status read command word is used to trigger the corresponding device to return the connection status. The connection status can be stored in the internal register of the CPLD chip for the CPU to read, or the CPLD can notify the CPU through GPIO. When the status of a certain NTB channel changes, the CPLD first completes the hardware-side status acquisition, and then provides the status to the CPU. The CPU then decides on the subsequent transmission channel adjustment, which is more timely than relying entirely on CPU software polling.

[0061] For example, when the first PCIe switch chip detects that the second NTB channel is disconnected, the first CPLD chip obtains this status through its local I2C channel and checks whether the backup controller also detects the same status via GPIO. If the statuses on both sides are consistent, the CPU can redirect the cached data to be synchronized on the second NTB channel to the first NTB channel, avoiding further data transmission to the faulty channel.

[0062] In a preferred embodiment, the first NTB channel and the second NTB channel are respectively configured with independent address mapping tables, BAR spaces and DMA channels;

[0063] The remote memory mapping region corresponding to the first NTB channel is physically isolated from the remote memory mapping region corresponding to the second NTB channel.

[0064] In this embodiment, the address mapping table records the correspondence between the local address and the accessible address of the peer. The BAR space exposes the accessible remote mapping window to the local system. The DMA channel is used for data transfer without relying on CPU byte-by-byte processing. Physical isolation of the remote memory mapping regions means that they do not share the same remote mapping window. If two types of NTB channels use the same mapping space, when the second NTB channel performs cached data synchronization, it may overwrite or over-access the area used for state interaction by the first NTB channel, resulting in abnormal connection state information or heartbeat detection information.

[0065] For example, the first NTB channel can map a smaller remote control area to store heartbeat flags, connection status words, and transaction log summaries; the second NTB channel can map a larger remote buffer area to transmit cached data to be synchronized. With these two types of areas separated, even if the second NTB channel performs a large-volume data transmission, it will not affect the heartbeat and status interaction of the first NTB channel.

[0066] In a preferred embodiment, the connection status information includes connectivity status, error status, and negotiation rate; the connectivity status indicates whether the first NTB channel and the second NTB channel are disconnected, the error status indicates whether the first NTB channel and the second NTB channel have encountered errors, and the negotiation rate indicates the data transmission rate of the first NTB channel and the second NTB channel.

[0067] In this embodiment, the connection status information includes at least three categories: connectivity status, error status, and negotiation rate. Connectivity status indicates whether the first or second NTB channel is in a communicable state. If a channel cannot maintain a PCIe connection, or if the two ends cannot exchange data through the corresponding NTB channel, the channel can be considered disconnected. Error status indicates whether transmission errors occurred during PCIe communication in the first or second NTB channel. Errors can originate from physical layer (PHY) errors, data layer errors, or transaction layer errors. Physical layer (PHY) errors typically reflect high-speed differential signal quality issues, data layer errors typically reflect packet verification or retransmission issues, and transaction layer errors typically reflect anomalies during PCIe transaction transmission. The negotiation rate represents the actual negotiated rate obtained by the first or second NTB channel after the PCIe connection is established. This rate can include the PCIe connection rate and the number of lanes. For example, if the system design aims for the second NTB channel to operate with higher bandwidth, but the actual negotiation result is lower or the number of lanes is reduced, the transmission capacity of the data to be synchronized and buffered will be decreased. In this embodiment, the connection status information simultaneously reflects whether the channel is disconnected, whether there are bit errors, and whether the current rate meets the transmission requirements. Based on this information, the CPU decides whether to continue using the original channel or switch to another NTB channel.

[0068] In a preferred embodiment, the first CPU and / or the second CPU are used to control the transmission channels of the cached data to be synchronized, the cross-controller I / O data, the connection status information, and the heartbeat detection information based on the connection status information.

[0069] In this embodiment, the control of the transmission channel includes three types of actions: maintaining the original division of labor, switching to another channel for transmission, and stopping the writing of new cached data to the buffer to be synchronized. When both the first NTB channel and the second NTB channel are available, the CPU maintains the original division of labor: connection status information and heartbeat detection information are transmitted via the first NTB channel, while the cached data to be synchronized and cross-controller I / O data are transmitted via the second NTB channel. When the second NTB channel is unstable, the CPU switches to the first NTB channel to transmit the necessary cached data to be synchronized. When the first NTB channel is unstable, the CPU switches to the second NTB channel to transmit connection status information and heartbeat detection information. The two NTB channels have different hardware sources and different carrying capacities; if one channel fails, the other channel can still provide backup communication as much as possible. The CPU adjusts the data path based on the status provided by the CPLD, which can avoid continuing to send critical data on the abnormal channel.

[0070] In a preferred embodiment, the method for the first CPU and / or the second CPU to control the data transmission channel based on connection status information includes:

[0071] When the first NTB channel is detected to be faulty and the second NTB channel is normal, the first CPU and the second CPU transmit the connection status information and the heartbeat detection information through the second NTB channel; when both the first NTB channel and the second NTB channel are faulty, the first CPLD chip and the second CPLD chip synchronize the NTB connection status of both parties and notify the first CPU and the second CPU respectively to perform fault isolation processing; the fault isolation processing includes stopping the writing of new cache data to the buffer to be synchronized.

[0072] In this embodiment, heartbeat detection information refers to the liveness confirmation information periodically exchanged between the first CPU and the second CPU, used to determine whether the peer controller is still in a communicable state. Heartbeat detection information may include heartbeat sequence number, timestamp, controller operating status flag, current channel flag, etc. Heartbeat detection information is control information; its data volume is small but it is sensitive to transmission delay, therefore it is transmitted by the first NTB channel when the first NTB channel is normal. The NTB connection status refers to the status information of the two NTB channels observed by the main controller and the backup controller respectively. Synchronization of the NTB connection status means that the first CPLD chip and the second CPLD chip keep the records of the two NTB channel statuses on the main controller and the backup controller consistent. Fault isolation processing means that when both the first and second NTB channels fail, the first CPU and the second CPU, based on the NTB connection status provided by the first and second CPLD chips, isolate the data path requiring cross-controller synchronization. Fault isolation processing can prevent cached data that needs to be synchronized to the peer controller from continuing to enter the synchronization buffer, avoiding further widening of the data difference between the main controller and the backup controller when both NTB channels are unavailable. Stopping the writing of new cached data to the buffer to be synchronized means no longer writing new dirty data to the buffer that needs to be synchronized across controllers, thus avoiding the continued generation of unsynchronized data when both channels fail.

[0073] In a preferred embodiment, the method for controlling the data transmission channel based on connection status information further includes: when the connection between the first NTB channel and the second NTB channel is restored to normal, the first CPU and the second CPU perform incremental data synchronization according to the transaction log, and the first NTB channel resumes transmitting the connection status information and the heartbeat detection information, and the second NTB channel resumes transmitting the cached data to be synchronized and the cross-controller IO data.

[0074] In this embodiment, during a channel anomaly, there may be data or state differences between the primary controller and the backup controller that have not been fully synchronized. Therefore, before resuming the original transmission division, incremental data synchronization needs to be performed based on the transaction log. The transaction log records transaction information related to cache synchronization before and after the anomaly, and may include write order, data block identifier, cache identifier, synchronization completion status, etc. Incremental data synchronization refers to only completing the synchronization of data that was not fully completed during the anomaly. The purpose of this process is to avoid directly switching back to the original division when the channel is restored, which would result in the data missed during the anomaly not being completed.

[0075] In a preferred embodiment, when the second NTB channel experiences bit errors, disconnections, or negotiation rates below the minimum rate threshold, and the first NTB channel does not experience bit errors, disconnections, or negotiation rates below the minimum rate threshold, the first CPU and the second CPU transmit the cached data to be synchronized through the first NTB channel.

[0076] In this embodiment, preferably, the first NTB channel is used to transmit the cached data to be synchronized required to maintain data consistency, and does not replace all the high-bandwidth services of the second NTB channel, prioritizing the synchronization of critical caches. Transmitting the cached data to be synchronized through the first NTB channel can maintain basic data consistency capabilities even with reduced bandwidth.

[0077] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0078] like Figure 2 As shown, in one embodiment, a transmission control method for a dual-controller memory interconnect is proposed, which may specifically include the following steps:

[0079] Step S10: Obtain the connection status information input by the hardware monitoring unit.

[0080] In this embodiment, the acquired connection status information refers to determining the status of the first NTB channel and the second NTB channel, specifically whether each NTB channel is disconnected, whether bit errors have occurred, and whether the negotiation rate is lower than the minimum rate threshold. This method can be executed by the first CPU and / or the second CPU. After acquiring the connection status information of the first and second NTB channels, the hardware monitoring unit inputs the connection status information to the corresponding CPU. After acquiring the connection status information, the CPU uses it to determine whether the two NTB channels can continue to handle data transmission under the current division of labor. The connection status information comes from a dedicated hardware monitoring unit, which can reduce CPU cycle usage and trigger subsequent transmission control faster when the PCIe connection status changes. The minimum rate threshold can be set according to system design requirements. For example, if the second NTB channel is used to transmit cached data to be synchronized and cross-controller I / O data, its minimum rate threshold can be higher than that of the first NTB channel. If the actual negotiation rate of the second NTB channel is lower than this threshold, even if it is not completely disconnected, it may not be able to meet the needs of cache synchronization and cross-controller I / O transmission.

[0081] Step S20: When neither the first NTB channel nor the second NTB channel has encountered any bit errors, disconnections, or negotiation rates below the minimum rate threshold, the connection status information and the heartbeat detection information are transmitted through the first NTB channel, and the data to be synchronized and the cross-controller I / O data are transmitted through the second NTB channel.

[0082] In this embodiment, when neither of the two NTB channels experiences bit errors or disconnection, and the negotiated rate is not lower than the corresponding minimum rate threshold, the system maintains normal hierarchical transmission. Control information is transmitted through the first NTB channel, and data information is transmitted through the second NTB channel, which reduces the impact of large-capacity data transmission on heartbeat detection and state synchronization.

[0083] Step S30: When the second NTB channel experiences bit errors, disconnection, or negotiation rate below the minimum rate threshold, and the first NTB channel does not experience bit errors, disconnection, or negotiation rate below the minimum rate threshold, the cached data to be synchronized is transmitted through the first NTB channel.

[0084] In this embodiment, if the first NTB channel is still available, the cached data to be synchronized will be transmitted through the first NTB channel to avoid the cached data to be synchronized being stuck when the second NTB channel is abnormal, and to maintain the synchronization of critical cached data between the main controller and the backup controller.

[0085] Step S40: When the first NTB channel experiences a bit error, disconnection, or negotiation rate below the minimum rate threshold, and the second NTB channel does not experience a bit error, disconnection, or negotiation rate below the minimum rate threshold, the connection status information and the heartbeat detection information are transmitted through the second NTB channel.

[0086] In this embodiment, if the second NTB channel is still available, the connection status information and heartbeat detection information are transmitted through the second NTB channel to avoid incorrect judgments by the main controller and the backup controller due to the loss of heartbeat detection information.

[0087] Step S50: When both the first NTB channel and the second NTB channel are disconnected, stop writing new cached data to the buffer to be synchronized.

[0088] In this embodiment, when both the first NTB channel and the second NTB channel are disconnected, the master controller and the backup controller can no longer reliably synchronize through the NTB channel, and at this time, writing new cached data to the buffer to be synchronized is stopped.

[0089] Step S60: After the connection between the first NTB channel and the second NTB channel is restored to normal, incremental synchronization of data between the master controller and the backup controller is performed based on the transaction log, and the transmission of the connection status information and the heartbeat detection information by the first NTB channel and the transmission of the cached data to be synchronized and the cross-controller IO data by the second NTB channel are restored.

[0090] In this embodiment, after the connection between the first NTB channel and the second NTB channel is restored, the CPU first performs incremental data synchronization based on the transaction log. The transaction log is used to record data that was not synchronized during the abnormal period. The CPU determines the range of data that needs to be synchronized in order to complete the data synchronization between the main controller and the backup controller based on the transaction log, and then completes the data synchronization between the main controller and the backup controller.

[0091] The advantage of the method provided in this embodiment is that the processing of this application separates the transmission of control information and data synchronization information. Heartbeat detection information and connection status information are not crowded out by data to be synchronized in the cache and cross-controller I / O data, thus preventing them from consuming channel resources. The hardware monitoring unit of this application provides the connection status, and the CPU selects the transmission channel based on this status. The transaction log is used for incremental synchronization after recovery, which can reduce the blocking caused by the mixed transmission of control information and large amounts of data, and reduce the risk of inconsistency between the dual-controller caches when the channel is abnormal.

[0092] In one embodiment, the transmission control method for dual-controller memory interconnect provided in this application can be implemented as a computer program that can run on a computer device. The computer program, consisting of various program steps, causes a processor to execute the steps in the transmission control method for dual-controller memory interconnect of the various embodiments of this application described in this specification.

[0093] It should be understood that although the steps in the flowcharts of the various embodiments of this application are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0094] Any references to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory.

[0095] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A dual-controller storage interconnect system, characterized in that, The system includes: Main controller, backup controller, first NTB channel, second NTB channel, and hardware monitoring unit; The main controller includes a first CPU and a first PCIe Switch chip; The backup controller includes a second CPU and a second PCIe switch chip; The first NTB channel includes a first NTB functional unit disposed in the first CPU, a second NTB functional unit disposed in the second CPU, and a first PCIe line group connecting the first NTB functional unit and the second NTB functional unit. The second NTB channel includes a first NTB port disposed in the first PCIe Switch chip, a second NTB port disposed in the second PCIe Switch chip, and a second PCIe line group connecting the first NTB port and the second NTB port; The first PCIe line group and the second PCIe line group are independent of each other; The hardware monitoring unit is used to acquire the connection status information of the first NTB channel and the second NTB channel, and send the connection status information to the first CPU and / or the second CPU. When both the first NTB channel and the second NTB channel are in normal connectivity, the first NTB channel is used to transmit connection status information and heartbeat detection information; the second NTB channel is used to transmit cached data to be synchronized and cross-controller I / O data.

2. The dual-controller storage interconnection system according to claim 1, characterized in that, The hardware monitoring unit includes a first CPLD chip disposed in the main controller and a second CPLD chip disposed in the backup controller; The first CPLD chip is connected to the first retimer chip and the first PCIe Switch chip in the main controller, and the first retimer chip is connected to the first CPU. The second CPLD chip is connected to the second retimer chip and the second PCIe switch chip in the backup controller, and the second retimer chip is connected to the second CPU; The first CPLD chip is used to obtain the connection status of the first NTB channel and the second NTB channel on the main controller side, and the second CPLD chip is used to obtain the connection status of the first NTB channel and the second NTB channel on the backup controller side.

3. The dual-controller storage interconnection system according to claim 2, characterized in that, The first CPLD chip sends a status read command word through the I2C channel to obtain the connection status of the first NTB channel and the second NTB channel on the main controller side, and obtains the connection status of the first NTB channel and the second NTB channel on the backup controller side through the GPIO channel. The second CPLD chip sends a status read command word through the I2C channel to obtain the connection status of the first NTB channel and the second NTB channel on the backup controller side, and obtains the connection status of the first NTB channel and the second NTB channel on the main controller side through the GPIO channel.

4. The dual-controller storage interconnection system according to claim 1, characterized in that, The first NTB channel and the second NTB channel are each configured with an independent address mapping table, BAR space and DMA channel; The remote memory mapping region corresponding to the first NTB channel is physically isolated from the remote memory mapping region corresponding to the second NTB channel.

5. A dual-controller storage interconnect system according to claim 2, characterized in that, The connection status information includes connectivity status, error rate, and negotiation rate. The connectivity status indicates whether the first NTB channel and the second NTB channel are disconnected, the bit error status indicates whether bit errors occur in the first NTB channel and the second NTB channel, and the negotiation rate indicates the data transmission rate of the first NTB channel and the second NTB channel.

6. The dual-controller storage interconnect system according to claim 5, characterized in that, The first CPU and / or the second CPU are used to control the transmission channels of the cached data to be synchronized, the cross-controller I / O data, the connection status information, and the heartbeat detection information based on the connection status information.

7. A dual-controller storage interconnection system according to claim 6, characterized in that, The methods by which the first CPU and / or the second CPU control the data transmission channel based on connection status information include: When the first NTB channel is detected to be faulty and the second NTB channel is normal, the first CPU and the second CPU transmit the connection status information and the heartbeat detection information through the second NTB channel. When both the first NTB channel and the second NTB channel fail, the first CPLD chip and the second CPLD chip synchronize the NTB connection status of both parties and notify the first CPU and the second CPU respectively to perform fault isolation processing. The fault isolation process includes stopping the writing of new cached data to the cache area to be synchronized.

8. A dual-controller storage interconnect system according to claim 7, characterized in that, Methods for controlling data transmission channels based on connection state information also include: Once the connection between the first NTB channel and the second NTB channel is restored to normal, the first CPU and the second CPU perform incremental data synchronization based on the transaction log, and enable the first NTB channel to resume transmitting the connection status information and the heartbeat detection information, and enable the second NTB channel to resume transmitting the cached data to be synchronized and the cross-controller IO data.

9. A dual-controller storage interconnection system according to claim 6, characterized in that, When the second NTB channel experiences a bit error, disconnection, or negotiation rate below the minimum rate threshold, and the first NTB channel does not experience a bit error, disconnection, or negotiation rate below the minimum rate threshold, the first CPU and the second CPU will transmit the cached data to be synchronized through the first NTB channel.

10. A transmission control method for dual-controller memory interconnection, characterized in that, The method is applied to the dual-controller memory interconnect system as described in any one of claims 1-9; The method includes: Obtain connection status information input from the hardware monitoring unit; When neither the first NTB channel nor the second NTB channel experiences bit errors, disconnections, or negotiation rates below the minimum rate threshold, the connection status information and the heartbeat detection information are transmitted through the first NTB channel, and the cached data to be synchronized and the cross-controller I / O data are transmitted through the second NTB channel. When the second NTB channel experiences bit errors, disconnections, or negotiation rates below the minimum rate threshold, and the first NTB channel does not experience bit errors, disconnections, or negotiation rates below the minimum rate threshold, the cached data to be synchronized is transmitted through the first NTB channel. When the first NTB channel experiences a bit error, disconnection, or negotiation rate below the minimum rate threshold, and the second NTB channel does not experience a bit error, disconnection, or negotiation rate below the minimum rate threshold, the connection status information and the heartbeat detection information are transmitted through the second NTB channel. When both the first NTB channel and the second NTB channel are disconnected, stop writing new cached data to the buffer to be synchronized. Once the connection between the first NTB channel and the second NTB channel is restored to normal, incremental synchronization of data between the primary controller and the backup controller is performed based on the transaction log. The connection status information and heartbeat detection information transmitted by the first NTB channel and the cached data to be synchronized and the cross-controller IO data transmitted by the second NTB channel are restored.

Images