Heartbeat communication method between host and backup based on PCIe NTB and related device

Abstract

The invention provides a PCIe NTB-based heartbeat communication method between a host and a standby, and a related device, and the method comprises the steps: building an NTB communication link between two hosts, so as to enable the two hosts to carry out heartbeat message transmission and user message transmission, and enabling one of the hosts to serve as a standby; in response to heartbeat message transmission or user message transmission, driving to newly build a sending task and putting the sending task into a sending work queue, and automatically scheduling the sending task by the kernel based on a preset scheduling strategy; different communication modes are selected for message sending, a message receiver selects a corresponding communication mode for message receiving, and the communication mode comprises DMA, a message window or a note; the user layer program is prompted to have a new user message based on the received message, and meanwhile, the user layer program acquires the heartbeat message at regular time. According to the method, heartbeat message transmission and user message transmission are carried out based on the NTB communication link, no extra data link path is occupied, and the method is suitable for the situation that two hosts are directly interconnected in a failover scene.

Description

Technical Field

[0001] The present application belongs to the technical field of PCIe switches, and in particular, relates to a heartbeat communication method and related devices based on a PCIe NTB master-slave relationship. Background Art

[0002] PCIe, the full name of Peripheral Component Interconnect Express, is a high-speed serial computer expansion bus standard. NTB (Non-Transparent Bridging) is a technology that allows host systems in different PCIe topology domains to communicate with each other. It enables a host to access devices and main memory in another PCIe topology domain through address mapping and translation. Communication between hosts can be achieved through NTB's cross-domain access to main memory. In an application scenario where multiple hosts are connected through NTB, when one host fails, another host can take over the service to achieve non-interruption failover switching.

[0003] At present, most of the existing methods for judging host failures use network-based heartbeat communication methods:

[0004] 1. The hosts on both sides are interconnected through network cards, which occupies additional communication links and may introduce new link problems or link configurations.

[0005] 2. Complex network usage methods, such as converting the NTB path into a virtual network path, have the advantage of embedding devices into the network to achieve cross-segment transmission. However, this application method requires a complex internal driver and is not suitable for failover scenarios where two hosts are directly connected.

[0006] 3. When two hosts communicate based on NTB, and heartbeat messages and user messages are transmitted separately. User message transmission will occupy communication resources. If there is a large amount of communication between the two hosts, it may cause the loss of heartbeat keep-alive messages, and eventually lead to abnormal failover switching. Similarly, heartbeat messages will also occupy communication resources when they are sent. If there is a large amount of communication between the two hosts, some user messages will not be sent, which will waste a lot of resources. Especially when using DMA for message transmission, since the CPU does not intervene when DMA sends messages, the bottleneck is often no longer the size of the packet but the number of packets, so the impact of DMA on transmission performance is particularly obvious. Summary of the invention

[0007] In view of this, the present application aims to propose a heartbeat communication method and related devices based on PCIe NTB master-slave to solve at least one of the above problems.

[0008] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0009] In a first aspect, the present application provides a heartbeat communication method between a PCIe NTB master and a standby, characterized in that it includes:

[0010] An NTB communication link is established between the two hosts so that the two hosts can transmit heartbeat messages and user messages, with one host acting as a backup.

[0011] In response to the transmission of heartbeat messages and / or user messages, the driver creates a new sending task and puts it into the sending work queue, and the kernel automatically schedules the sending task based on a preset scheduling policy;

[0012] Select different communication modes to send messages, and the message receiver selects the corresponding communication mode to receive messages, wherein the communication mode includes DMA, message window or note;

[0013] Based on the received message, the user layer program is notified that there is a new user message, and the user layer program obtains the heartbeat message at a regular interval.

[0014] In the second aspect, based on the same inventive concept, the present application also provides a heartbeat communication device based on a PCIe NTB master-slave connection, including:

[0015] A link building module is configured to establish an NTB communication link between two hosts so that the two hosts can perform heartbeat message transmission and user message transmission, with one host serving as a standby;

[0016] The task scheduling module is configured to drive the creation of a new sending task and put it into the sending work queue in response to the heartbeat message transmission and / or user message transmission, and the kernel automatically schedules the sending task based on a preset scheduling strategy;

[0017] The transmission module is configured to select different communication modes to send messages, and the message receiver selects the corresponding communication mode to receive messages, wherein the communication mode includes DMA, message window or note;

[0018] The message acquisition module is configured to prompt the user layer program that there is a new user message based on the received message, and the user layer program obtains the heartbeat message regularly.

[0019] In a third aspect, based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0020] In a fourth aspect, based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0021] Compared with the prior art, the heartbeat communication method and related device based on PCIe NTB master-slave described in this application have the following beneficial effects:

[0022] The heartbeat communication method and related device based on PCIe NTB master-slave described in the present application perform heartbeat message transmission and user message transmission based on the constructed NTB communication link, do not occupy additional data link paths, effectively ensure the transmission performance of DMA, and are suitable for situations where two hosts are directly connected in a failover scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 This is a flow chart of the heartbeat communication method between the PCIe NTB master and standby according to an embodiment of the present application;

[0025] Figure 2 This is a schematic diagram of message communication transmission between two hosts according to an embodiment of the present application;

[0026] Figure 3 A schematic diagram of message construction when two hosts perform message transmission according to an embodiment of the present application;

[0027] Figure 4 This is a schematic diagram of the structure of a heartbeat communication device based on PCIe NTB master-slave according to an embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of the hardware structure of the electronic device described in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] See also Figure 1 and Figure 2 As shown, this embodiment provides a heartbeat communication method between a PCIe NTB master and a standby, which specifically includes the following steps:

[0033] Step S101, establishing an NTB communication link between two hosts so that the two hosts can transmit heartbeat messages and user messages, with one host serving as a standby;

[0034] Specifically, in this embodiment, the present application performs heartbeat message transmission and user message transmission based on the NTB communication link, without occupying an additional data link path. Specifically, a link relationship between two hosts is established based on the NTB doorbell register, and a message window channel between two hosts is established based on the NTB note register (the note register is set to allow message transmission between the two hosts) and the BAR (Base Address Register, BAR is configured on the NT interface of each host, BAR defines the base address of the memory area, and these addresses will be mapped to the address space of another host) of the NT interface, and a DMA channel is established according to the message window capability and DMA capability, thereby establishing an NTB communication link with message transmission.

[0035] When the NTB link is successfully established, a character device is established in each of the two host domains to provide user-level message sending / receiving interface, memory mapping interface and control interface, such as write / read / mmap / ioctl / poll interfaces. The implementation is simple, a unified file system is used, and the user is convenient to use.

[0036] Step S102: In response to the heartbeat message transmission and / or user message transmission, the driver creates a new sending task and puts it into the sending work queue, and the kernel automatically schedules the sending task based on a preset scheduling policy.

[0037] Specifically, in this embodiment, Figure 3 As shown, when two hosts communicate based on NTB, a 4-byte message header information is reserved when transmitting messages in the driver, including payload length (if it is a heartbeat message, the payload length is zero; if it is a user message, the payload length is the user message length), CRC check, heartbeat transmission and other information.

[0038] When there is a heartbeat message transmission, the driver creates a new heartbeat sending task and puts it into the sending work queue. When there is a user message to be sent, a new message sending task is created and put into the sending work queue.

[0039] The kernel automatically schedules tasks in the send work queue based on the preset scheduling policy, as follows:

[0040] In response to the trigger message sending task, check whether there is a heartbeat message sending task in the sending work queue. If there is, the heartbeat information is carried in the reserved message header, and the heartbeat sending task is canceled at the same time; if there is no heartbeat message, the user message is sent directly;

[0041] In response to triggering the heartbeat sending task, the sending work queue is traversed to see if there is a user message sending task. If so, the heartbeat message is carried in the message header reserved for the user message, and the user message sending task is canceled at the same time; if not, the heartbeat message is sent directly.

[0042] In addition, when the kernel automatically schedules the sending task, it also detects other tasks in the sending task. The heartbeat message user ensures the keep-alive function of the hosts on both sides, and its information is filled in the 4-byte message header; the user message carrying the user load can choose to block and wait for a response (for example: after sending a message, the sender will suspend the execution of the current task until receiving a response from the receiver), or it can not block the sending (for example: after sending a message, the sender will not wait for a response and continue to execute other tasks).

[0043] This embodiment can send user messages through write at the same time as sending the heartbeat. The two can be transmitted simultaneously at the driver layer and do not affect each other at the user layer, making full use of the transmission capacity of the NTB channel and the transmission capacity of the DMA.

[0044] Step S103: Select different communication modes to send messages, and the message receiver selects a corresponding communication mode to receive messages, wherein the communication modes include DMA, message window or note.

[0045] Specifically, in this embodiment, the corresponding communication mode is selected according to the capability of the message window and the DMA capability;

[0046] In response to the DMA capability, the MDA is preferentially selected for message transmission;

[0047] In response to the lack of DMA capability and the availability of message window capability, the message window is selected for message transmission;

[0048] In response to the lack of DMA capability and message window capability, a note is selected for fragmented transfer.

[0049] Step S104: Based on the received message, the user layer program is notified that there is a new user message, and the user layer program obtains the heartbeat message at a regular interval.

[0050] Specifically, in this embodiment, the doorbell is used to notify the other party that a message is sent to the other party, and the other party is informed of the communication method;

[0051] After receiving the doorbell, the message receiver determines the communication mode according to the doorbell type. If it is a note communication, the message is received through the note. If it is DMA or a message window, the message is received from the message window. The driver notifies the user-layer program of new user messages through poll / read, etc. At the same time, the user-layer program obtains heartbeat messages regularly through ioctl.

[0052] The heartbeat communication method described in this embodiment is based on user layer interfaces such as write / read / mmap / poll provided by character devices, is simple and convenient to use, and uses ioctl to manage heartbeat and configuration information, etc.; while sending the heartbeat, user messages can be sent through write at the same time, and the two can be transmitted simultaneously at the driver layer, and do not affect each other at the user layer, making full use of the transmission capacity of the NTB channel. At the same time, the heartbeat message and the user message are all transmitted based on the NTB channel, without occupying additional physical links, effectively ensuring the transmission performance of DMA, and are suitable for the situation where two hosts are directly connected in the failover scenario.

[0053] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0054] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application further provides a heartbeat communication device based on a PCIe NTB master-slave relationship.

[0055] like Figure 4 As shown, the heartbeat communication device based on PCIe NTB master-slave includes:

[0056] The link building module 11 is configured to establish an NTB communication link between two hosts so that the two hosts can perform heartbeat message transmission and user message transmission, and one of the hosts serves as a standby;

[0057] The task scheduling module 12 is configured to drive a new sending task and put it into the sending work queue in response to the heartbeat message transmission or user message transmission, and the kernel automatically schedules the sending task based on a preset scheduling strategy;

[0058] The transmission module 13 is configured to select different communication modes to send messages, and the message receiver selects the corresponding communication mode to receive messages, wherein the communication mode includes DMA, message window or note;

[0059] The message acquisition module 14 is configured to prompt the user layer program that there is a new user message based on the received message, and the user layer program obtains the heartbeat message regularly.

[0060] For the convenience of description, the above devices are described in terms of functions and are divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0061] The device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0062] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0063] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0064] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0065] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0066] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0067] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0068] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0069] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0070] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0071] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0072] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0073] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0074] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0075] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0076] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0077] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A heartbeat communication method between PCIe NTB master and standby, characterized in that: include: An NTB communication link is established between the two hosts so that the two hosts can transmit heartbeat messages and user messages, with one host acting as a backup. In response to the transmission of heartbeat messages and / or user messages, the driver creates a new sending task and puts it into the sending work queue, and the kernel automatically schedules the sending task based on a preset scheduling policy; Select different communication modes to send messages, and the message receiver selects the corresponding communication mode to receive messages, wherein the communication mode includes DMA, message window or note; Based on the received message, the user layer program is notified that there is a new user message, and the user layer program obtains the heartbeat message at a regular interval.

2. The method according to claim 1, characterized in that The step of establishing an NTB communication link between two hosts so that the two hosts can perform heartbeat message transmission and user message transmission includes: Build the link relationship between the two hosts based on the NTB doorbell register, and build the NTB message window channel and DMA channel to establish the NTB communication link; A virtual character device is created in each of the two host domains, wherein the character device provides a user layer message sending / receiving interface, a memory mapping interface and a control interface.

3. The method according to claim 1, characterized in that Also includes: When transmitting messages in the driver, message header information is reserved, and the message header information at least includes payload length, CRC check, and heartbeat transmission.

4. The method according to claim 1, characterized in that The scheduling strategy includes: In response to the triggering of the message sending task, it traverses the sending work queue to see if there is a heartbeat message sending task. If there is, the heartbeat information is carried in the reserved message header, and the heartbeat sending task is canceled at the same time; if there is no heartbeat message, the user message is sent directly; In response to triggering the heartbeat sending task, the sending work queue is traversed to see if there is a user message sending task. If so, the heartbeat message is carried in the message header reserved for the user message, and the user message sending task is canceled at the same time; if not, the heartbeat message is sent directly.

5. The method according to claim 1, characterized in that The selecting of different communication modes to send messages, and the message receiver selecting a corresponding communication mode to receive messages, wherein the communication mode includes DMA, message window or note, including: Select the corresponding communication method according to the capabilities of the message window and DMA capabilities; In response to the DMA capability, the MDA is preferentially selected for message transmission; In response to the lack of DMA capability and the availability of message window capability, the message window is selected for message transmission; In response to the lack of DMA capability and message window capability, a note is selected for fragmented transfer.

6. The method according to claim 5, characterized in that Also includes: Notify the other party through the doorbell that a message has been sent to the other party, and inform the other party of the communication method; After receiving the doorbell, the message receiver determines the communication mode according to the doorbell type. If it is a note communication, the message is received through the note. If it is DMA or a message window, the message is received from the message window.

7. A heartbeat communication device based on PCIe NTB master-slave, characterized in that: include: A link building module is configured to establish an NTB communication link between two hosts so that the two hosts can perform heartbeat message transmission and user message transmission, with one host serving as a standby; The task scheduling module is configured to drive the creation of a new sending task and put it into the sending work queue in response to the heartbeat message transmission and / or user message transmission, and the kernel automatically schedules the sending task based on a preset scheduling strategy; The transmission module is configured to select different communication modes to send messages, and the message receiver selects the corresponding communication mode to receive messages, wherein the communication mode includes DMA, message window or note; The message acquisition module is configured to prompt the user layer program that there is a new user message based on the received message, and the user layer program obtains the heartbeat message regularly.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: in, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

Images