Multi-node memory synchronization method and device, equipment, storage medium and program product

Abstract

The invention relates to a multi-node memory synchronization method and device, equipment, a storage medium and a program product. The method comprises the following steps: establishing connection among nodes in a distributed system, configuring local / far-end queues for recording the busy degree of a thread for each node, creating a queue pair associated with a unique doorbell address for each node in combination, and optimizing a retention rule and an error reporting mechanism of a working queue element (WQE) in the queue pair, so that the busy degree of the thread is recorded; batch synchronization of multi-node memory data is achieved by adding an RDMA write operation WQE recording the busy state of a thread before a far-end mapping memory area is newly added and triggering hardware to execute RDMA write operations corresponding to all WQEs in a queue pair in sequence when the busy state of a local thread is updated. By the adoption of the method, the switching frequency of a CPU core between the synchronization task and the calculation task can be reduced, efficient, low-overhead and high-reliability multi-node memory synchronization is achieved, and the method meets the performance requirement of a large-scale distributed system.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a multi-node memory synchronization method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, Remote Direct Memory Access (RDMA) technology has emerged. While existing RDMA-based distributed systems can alleviate data transmission blindness and ensure basic stability through a dual-buffer mechanism (local / remote subqueries separately recording thread busy levels), they exhibit significant shortcomings in multi-node memory synchronization scenarios. The current mainstream model of "detecting local data updates and then initiating RDMA write operations node by node" requires repeatedly preparing work queue elements (WQE) and executing post-send function calls for each operation. This not only leads to complex software logic and susceptibility to errors but also consumes significant CPU resources. Therefore, the existing RDMA connection buffer mechanism cannot meet the efficiency requirements of multi-node memory synchronization. Summary of the Invention

[0003] Therefore, it is necessary to provide a multi-node memory synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can meet the high efficiency requirements of multi-node memory synchronization in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a multi-node memory synchronization method, including:

[0005] Establish connections between nodes in the distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0006] Create a queue pair for each node and associate a unique doorbell address with the queue pair. The queue pair is configured to be retained in the queue pair after the work queue element is executed, and the completed queue element is reported to the completion queue when an execution error occurs.

[0007] When a remote mapped memory region is added to the local memory region of a node, before the actual data synchronization, a corresponding RDMA write operation work queue element is added to the queue pair. The work queue element defines the thread busy status of the local memory region to be written to the newly added remote mapped memory region.

[0008] When the thread busy state in the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair to trigger the hardware to execute the RDMA write operation corresponding to all work queue elements in the queue pair in sequence, and synchronize the updated local memory region data to all associated remote mapped memory regions to complete multi-node memory synchronization.

[0009] Secondly, this application also provides a multi-node memory synchronization device, comprising:

[0010] Create a module to establish connections between nodes in a distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0011] The creation module is also used to create queue pairs for each node and associate a unique doorbell address with each queue pair. The queue pairs are configured to be retained in the queue pair after the work queue element is executed, and to report the completed queue element to the completion queue when an execution error occurs.

[0012] The write module is used to add a corresponding RDMA write operation work queue element to the queue pair before actual data synchronization when a remote mapped memory region is added to the local memory region of a node. The work queue element defines the thread busy state of the local memory region to be written to the newly added remote mapped memory region.

[0013] The trigger module is used to write a preset trigger flag to the doorbell address associated with the queue pair when the thread busy state in the local memory area is updated. This triggers the hardware to execute the RDMA write operations corresponding to all work queue elements in the queue pair in sequence, and synchronizes the updated local memory area data to all associated remote mapped memory areas, thus completing multi-node memory synchronization.

[0014] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0015] Establish connections between nodes in the distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0016] Create a queue pair for each node and associate a unique doorbell address with the queue pair. The queue pair is configured to be retained in the queue pair after the work queue element is executed, and the completed queue element is reported to the completion queue when an execution error occurs.

[0017] When a remote mapped memory region is added to the local memory region of a node, before the actual data synchronization, a corresponding RDMA write operation work queue element is added to the queue pair. The work queue element defines the thread busy status of the local memory region to be written to the newly added remote mapped memory region.

[0018] When the thread busy state in the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair to trigger the hardware to execute the RDMA write operation corresponding to all work queue elements in the queue pair in sequence, and synchronize the updated local memory region data to all associated remote mapped memory regions to complete multi-node memory synchronization.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0020] Establish connections between nodes in the distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0021] Create a queue pair for each node and associate a unique doorbell address with the queue pair. The queue pair is configured to be retained in the queue pair after the work queue element is executed, and the completed queue element is reported to the completion queue when an execution error occurs.

[0022] When a remote mapped memory region is added to the local memory region of a node, before the actual data synchronization, a corresponding RDMA write operation work queue element is added to the queue pair. The work queue element defines the thread busy status of the local memory region to be written to the newly added remote mapped memory region.

[0023] When the thread busy state in the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair to trigger the hardware to execute the RDMA write operation corresponding to all work queue elements in the queue pair in sequence, and synchronize the updated local memory region data to all associated remote mapped memory regions to complete multi-node memory synchronization.

[0024] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0025] Establish connections between nodes in the distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0026] Create a queue pair for each node and associate a unique doorbell address with the queue pair. The queue pair is configured to be retained in the queue pair after the work queue element is executed, and the completed queue element is reported to the completion queue when an execution error occurs.

[0027] When a remote mapped memory region is added to the local memory region of a node, before the actual data synchronization, a corresponding RDMA write operation work queue element is added to the queue pair. The work queue element defines the thread busy status of the local memory region to be written to the newly added remote mapped memory region.

[0028] When the thread busy state in the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair to trigger the hardware to execute the RDMA write operation corresponding to all work queue elements in the queue pair in sequence, and synchronize the updated local memory region data to all associated remote mapped memory regions to complete multi-node memory synchronization.

[0029] The aforementioned multi-node memory synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program products, by configuring local / remote subqueues to record thread busyness for each node in the distributed system, combined with creating a queue pair associated with a unique doorbell address for each node, and optimizing the retention rules and error reporting mechanism of work queue elements in the queue pair, ensure the accuracy of thread state awareness between nodes and the traceability of operational anomalies. When a remote mapped memory region is added to the local memory region of a node, a work queue element corresponding to the RDMA write operation is added to the queue pair before actual data synchronization. When the thread busyness status of the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair, triggering the hardware to execute the RDMA write operations corresponding to all work queue elements in the queue pair in sequence, synchronizing the updated local memory data to all associated remote mapped memory regions, thereby completing multi-node memory synchronization. This significantly simplifies the software operation logic, reduces the CPU resource consumption caused by post-send function calls, reduces the frequency of CPU core switching between synchronization and computation tasks, and achieves efficient, low-overhead, and highly reliable multi-node memory synchronization, adapting to the performance requirements of large-scale distributed systems. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1This is an application environment diagram of a multi-node memory synchronization method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a multi-node memory synchronization method in one embodiment;

[0033] Figure 3 This is a flowchart of a multi-node memory synchronization method in one embodiment;

[0034] Figure 4 This is a flowchart illustrating the tool interaction process of a multi-node memory synchronization method in one embodiment.

[0035] Figure 5 This is a schematic diagram illustrating the initialization of the memory synchronization QP when a new connection is established between the local server and the remote server in one embodiment.

[0036] Figure 6 This is a schematic diagram of the update phase of a multi-node memory synchronization method in one embodiment;

[0037] Figure 7 This is a structural block diagram of a multi-node memory synchronization device in one embodiment;

[0038] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0041] The multi-node memory synchronization method provided in this application can be applied to, for example... Figure 1The application environment shown can involve only terminal 102, only server 104, or both terminal 102 and server 104. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Specifically, terminal 102 or server 104 implements a multi-node memory synchronization method. This method configures local / remote subqueries to record thread busy levels for each node in the distributed system. Combined with creating queue pairs associated with unique doorbell addresses for each node, and optimizing the retention rules and error reporting mechanisms for work queue elements in the queue pairs, it ensures the accuracy of thread state awareness between nodes and the traceability of operational anomalies. When a remote mapped memory region is added to a node's local memory region, a work queue element corresponding to the RDMA write operation is added to the queue pair before actual data synchronization. When the thread busy state of the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair, triggering the hardware to execute the RDMA write operations corresponding to all work queue elements in the queue pair in sequence. This synchronizes the updated local memory data to all associated remote mapped memory regions, thereby completing multi-node memory synchronization. This significantly simplifies the software operation logic, reduces the CPU resource consumption caused by post and send function calls, and reduces the frequency of CPU core switching between synchronization and computation tasks. It achieves efficient, low-overhead, and highly reliable multi-node memory synchronization, adapting to the performance requirements of large-scale distributed systems.

[0042] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0043] In one exemplary embodiment, such as Figure 2 As shown, a multi-node memory synchronization method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0044] Step 202: Establish connections between nodes in the distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0045] The core of this step is to establish the basic communication infrastructure between nodes in the distributed system and configure the state awareness carrier. First, the "establish" action establishes communication links between all nodes, which is the physical foundation for subsequent memory synchronization data transmission. Next, for each established inter-node connection, the "configure" action deploys two types of sub-queues. The local sub-queue, as the native queue, directly records the busy status of the local node's threads and serves as the source state data carrier for memory synchronization. The remote sub-queue, relying on a "mapping" mechanism, is associated with the local sub-queue of the remote node and specifically records the busy status of the remote threads. This allows the local node to directly perceive the peer's status, avoiding the problem of blindly sending packets due to unknown peer status in traditional transmission, thus laying the foundation for the accuracy of subsequent synchronization operations.

[0046] The local sub-queue for recording the busyness of local threads typically records dimensions such as thread ID, current thread status (busy / idle / load percentage), and status update timestamp, for example, "Thread 2 of node A has an occupancy rate of 80% (busy)" and "Thread 5 of node A is idle".

[0047] Step 204: Create a queue pair for each node and associate a unique doorbell address with the queue pair. The queue pair is configured to be retained in the queue pair after the work queue element is executed, and the completed queue element is reported to the completion queue when an execution error occurs.

[0048] This step focuses on the customized creation and configuration of the RDMA operation execution carrier. First, a dedicated queue pair (QP) is created for each node in the system. This queue pair is the core carrier for submitting and executing RDMA write operation requests, distinct from traditional general-purpose QPs, providing a dedicated execution channel for multi-node synchronous operations. Then, a unique doorbell address is associated with this queue pair, establishing a unique correspondence between software trigger signals and hardware execution actions, ensuring the accuracy of trigger signals and avoiding signal confusion between different nodes and different queue pairs. Simultaneously, the core attributes of the queue pair are configured: on the one hand, a rule is set that work queue elements (WQEs) are not destroyed after execution but remain in the queue pair, breaking the traditional "execution equals destruction" model and enabling WQE reuse; on the other hand, completion queue elements (CQEs) are limited to reporting to the completion queue only when WQE execution fails, eliminating redundant feedback in normal scenarios and reducing invalid processing at the software level.

[0049] The system comprises queue pairs (QPs), including receive queues (RQs) and send queues (SQs), used by software to submit receive buffers and send requests to hardware. Each QP is numbered as QPN. During QP creation, its attributes are configured according to the requirements of this invention, including associating it with a unique doorbell address. Work queue elements (WQEs) are send requests within a QP, populated by software and consumed by hardware. Completion queue elements (CQEs) are completion requests within a QP, populated by hardware and consumed by software. The system continuously monitors the execution status of WQEs within a QP, generating a CQE report only when an error occurs.

[0050] Step 206: In the case of adding a remote mapped memory region to the local memory region of a node, before the actual data synchronization, add a corresponding RDMA write operation work queue element to the queue pair. The work queue element defines that the thread busy status of the local memory region is written to the newly added remote mapped memory region.

[0051] This step is a preparatory stage for synchronization operations, the core of which lies in decoupling the configuration of RDMA write operations from the real-time synchronization sequence. When the node's local memory region encounters the trigger condition of "adding" a remote mapped memory region, the processing does not wait until the actual data synchronization phase. Instead, the corresponding RDMA write operation WQE is "added" to a dedicated queue in advance. This WQE is explicitly "defined" as carrying the operation instruction of "writing the newly added remote mapped memory region when the local memory region thread is busy." It belongs to the control plane preparation work and is not bound to the real-time requirements of data transmission. This design avoids the complex process of temporarily creating and configuring WQEs during traditional synchronization, reduces the software intervention steps when synchronization is triggered, and improves subsequent synchronization efficiency.

[0052] Remote Direct Memory Access (RDMA) is a technology that allows data transfer directly between computer memory locations. It enables one computer system (local node) to directly access the memory of another computer system (remote node) without the need for multiple data copies and complex buffering processes by the operating system kernel, as is the case with traditional network communication.

[0053] Step 208: When the thread busy state in the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair to trigger the hardware to execute the RDMA write operation corresponding to all work queue elements in the queue pair in sequence, and synchronize the updated local memory region data to all associated remote mapped memory regions to complete multi-node memory synchronization.

[0054] This step is the execution phase of multi-node memory synchronization. When the thread busy state of the local memory region is "updated," the synchronization process is triggered: the software "writes" a preset trigger flag to the doorbell address associated with the queue pair. This operation is essentially sending an execution signal to the hardware. After receiving the signal, the hardware "executes" the RDMA write operations corresponding to all configured WQEs in the queue pair in sequence. This eliminates the need for the software to initiate write requests to each remote node individually, directly synchronizing the updated local thread busy state to all associated remote mapped memory regions. The entire process relies on the pre-configured WQEs and dedicated queue pairs, requiring only a single doorbell trigger operation to complete the synchronization of all remote nodes. This replaces the complex process of traditionally traversing each node and initiating RDMA writes sequentially, ultimately achieving efficient multi-node memory synchronization.

[0055] In the aforementioned multi-node memory synchronization method, by configuring local / remote subqueries to record thread busy levels for each node in the distributed system, combined with creating a queue pair associated with a unique doorbell address for each node, and optimizing the retention rules and error reporting mechanism of work queue elements in the queue pair, the accuracy of thread state awareness between nodes and the traceability of operational anomalies are ensured. When a remote mapped memory region is added to the local memory region of a node, a work queue element corresponding to the RDMA write operation is added to the queue pair before actual data synchronization. When the thread busy status of the local memory region is updated, a preset trigger flag is written to the doorbell address associated with the queue pair, triggering the hardware to execute the RDMA write operations corresponding to all work queue elements in the queue pair in sequence, synchronizing the updated local memory data to all associated remote mapped memory regions, thereby completing multi-node memory synchronization. This significantly simplifies the software operation logic, reduces the CPU resource consumption caused by post-send function calls, reduces the frequency of CPU core switching between synchronization and computation tasks, and achieves efficient, low-overhead, and highly reliable multi-node memory synchronization, adapting to the performance requirements of large-scale distributed systems.

[0056] In one embodiment, a queue pair is created for each node, and a unique doorbell address is associated with each queue pair, including:

[0057] During system initialization, each node is allocated a queue pair resource for multi-node memory synchronization;

[0058] Configure the communication mode of the queue pair as reliable connectionless mode, configure the retention policy of the work queue elements in the queue pair as not to delete after execution and to retain them for reuse in the next synchronization, and configure the reporting policy of the queue elements as to only report when the work queue element execution fails.

[0059] Assign a unique doorbell address to each queue pair and establish a mapping relationship between the doorbell address and the queue pair.

[0060] The "during system initialization" setting limits the timing of resource allocation, bringing queue pair creation forward to the system startup phase to avoid performance degradation and process delays caused by dynamic resource allocation during runtime. "Allocating to each node" clarifies the scope of resource allocation, ensuring each node has its own dedicated queue pair resource, not shared with other nodes, thus avoiding resource contention. "Used for multi-node memory synchronization" defines the specific purpose of the queue pair, distinguishing it from other general-purpose queue pairs in the system. This ensures that all configurations and operations of this queue pair are optimized for multi-node memory synchronization scenarios, guaranteeing the independence and efficiency of synchronization operations at the resource level. Through this process, the timing and purpose of queue pair resource allocation are clearly defined.

[0061] Next, the core attributes of the queues can be finely configured, focusing on the customization of communication modes and feedback rules. First, the communication mode of the queues is configured as "reliable connectionless mode." This mode eliminates the need to establish a dedicated connection path for each remote synchronization target, significantly reducing the overhead and complexity of connection management and adapting to large-scale scenarios with multiple nodes. Second, a retention policy is configured for work queue elements (WQEs) to "not delete after execution but retain for reuse in the next synchronization," breaking the traditional WQE execution-and-destroy model and avoiding the repeated creation and filling of WQEs with each synchronization, thus reducing CPU resource consumption. Finally, a strategy is configured for completion queue elements (CQEs) to "report only when an error occurs during the execution of a work queue element," eliminating redundant feedback in normal synchronization scenarios, allowing the system to handle only genuine error situations, reducing ineffective processing costs at the software level, and improving system stability.

[0062] Finally, "assigning a unique doorbell address to each queue pair" ensures that each queue pair has a dedicated hardware trigger signal entry, avoiding confusion of trigger signals between different nodes and different queue pairs; "establishing a mapping relationship between doorbell addresses and queue pairs" strongly binds the trigger signal to the execution carrier. When the software writes a trigger identifier to the doorbell address, the hardware can accurately locate the corresponding queue pair and execute the WQE within it, ensuring the accuracy and efficiency of the trigger operation and avoiding synchronization errors or missed execution problems caused by signal mismatch.

[0063] During system startup, in this embodiment of the invention, a doorbell address corresponding to a multi-node memory synchronization mechanism is created for each node. During the creation of the QP, its attributes are configured according to the requirements of this invention, including associating it with a unique doorbell address. For each node, all remote nodes that need to be synchronized with its memory are first identified, and a WQE corresponding to the RDMA write operation that synchronizes the local memory region to the remote target memory region is added to the QP.

[0064] In the above embodiments, by pre-allocating dedicated multi-node memory synchronization queue pairs to each node during the system initialization phase, combined with the configuration of a reliable connectionless communication mode, the connection management overhead in multi-node synchronization scenarios is reduced. Simultaneously, by explicitly defining the retention strategy for WQE reuse after execution and the feedback strategy of CQE only reporting errors, the CPU overhead of repeatedly creating WQE and handling redundant feedback at the software level is further reduced. Furthermore, the precise mapping between unique doorbell addresses and queue pairs ensures the uniqueness of trigger signals and the accuracy of execution. These features, along with features such as dual-queue state awareness, pre-configured WQE, and batch execution of doorbell triggers, work together to further improve the efficiency and reliability of multi-node memory synchronization, reduce overall system resource consumption and software complexity, and better adapt to the synchronization needs of large-scale distributed systems.

[0065] In one embodiment, the implementation process of doorbell address-triggered synchronization includes:

[0066] If the hardware detects a change in the data in the doorbell address in real time, and the change is a preset trigger identifier, it determines that a synchronization request has been received. The hardware then directly reads all configured work queue elements from the queue pair and starts the RDMA write operation execution process.

[0067] First, it's crucial to clarify that the triggering prerequisite is "real-time hardware monitoring" of the doorbell address's data status. Unlike traditional software-level polling or passive responses, hardware-level real-time monitoring significantly reduces the recognition latency of trigger signals. The core judgment condition for monitoring is that "the data in the doorbell address changes and changes to a preset trigger identifier." This dual condition avoids invalid triggers caused by erroneous data changes and ensures that only signals conforming to preset rules can activate the synchronization process, guaranteeing the accuracy of the triggering operation. When the above conditions are met, the hardware directly "determines that a synchronization request has been received," eliminating the need for software-level relay judgment and further reducing intervention steps. Subsequently, the hardware "directly reads all configured work queue elements from the queue pair" and "initiates the RDMA write operation execution process." The entire process is completed by the hardware, skipping the complex steps of software traversing remote nodes one by one and initiating RDMA write requests sequentially in traditional synchronization. It relies on pre-configured WQE to achieve batch execution, maximizing the technological advantages of RDMA hardware direct access to memory.

[0068] For example, such as Figure 5As shown, in the distributed system consisting of N nodes, node 2 updates its local busy state buffer, and the gray area changes from 2 to 1, indicating that memory region 2 is no longer busy, while 1 is busy. At this time, a specific value is written to its doorbell address according to the QP (QP2) corresponding to node 2's memory synchronization mechanism. Upon receiving the notification, the hardware immediately begins executing the WQE for all RDMA writes in that QP, quickly synchronizing the updated data to all relevant remote nodes. At this point, position 1 in all remote mapped memory regions connected to node 2 from node 1 to node N is updated to busy. In this process, since the WQE is pre-configured during initialization and will always remain in the QP corresponding to the memory synchronization mechanism, only doorbell synchronization needs to be performed. Therefore, complex software reconfiguration and numerous function calls are unnecessary, greatly improving synchronization efficiency and reducing the latency from data update to synchronization completion.

[0069] The above embodiments clarify the specific implementation methods of real-time hardware monitoring, accurate determination of trigger conditions, direct reading of WQE and execution of RDMA write operations, and deep collaboration with features such as dual-queue state awareness, dedicated queue pair configuration, and pre-added WQE: hardware-level real-time monitoring and accurate determination reduce the recognition delay and false trigger probability of trigger signals, while direct hardware reading and execution of WQE completely eliminates software intervention in the synchronous execution process, further reducing CPU resource consumption and software operation complexity; at the same time, combined with the design of WQE reuse and batch execution, the response speed and execution efficiency of multi-node memory synchronization are greatly improved, and the hardware-led execution method also enhances the stability of the synchronization process, effectively solving the problems of high latency, large overhead, and easy error caused by traditional software-led synchronization, and further strengthening the efficiency and reliability of memory synchronization in large-scale distributed systems.

[0070] In one embodiment, the association process between the local memory region and the remote mapped memory region includes:

[0071] When a connection is established between a local node and a remote node, the registration information of the memory regions of both parties is exchanged through the RDMA protocol. The registration information includes virtual address, memory length and remote access key.

[0072] Based on the registration information of the received remote memory region, a remote mapping memory region is established on the local node. The remote mapping memory region is a mirror mapping of the local sub-queue of the remote node.

[0073] Record the correspondence between the local memory region and the remote mapped memory region, and configure the source address and destination address in the work queue element based on the correspondence, so as to synchronously add the corresponding work queue element to the queue pair.

[0074] The above process clarifies the complete workflow for establishing a connection between the local memory region and the remote mapped memory region. The core is relying on the RDMA protocol to complete memory information exchange and mapping construction, providing address information for subsequent WQE configuration. During the connection establishment phase between the local and remote nodes, the registration information of both memory regions is first exchanged via the RDMA protocol. This information includes core content such as virtual address, memory length, and remote access key, which is the foundation for cross-node direct memory access. Next, based on the received remote registration information, the local node creates a remote mapped memory region. This region does not independently store data but is a mirror mapping of the remote node's local sub-queue, ensuring accurate association and synchronization of the remote node's thread busy state. Finally, the correspondence between the local and remote mapped memory regions is recorded, and the source (local memory region) and destination (remote mapped memory region) addresses in the WQE are configured according to this relationship. This provides a precise address configuration basis for synchronously adding the corresponding WQE to the queue pair when adding a new remote mapped memory region, ensuring that the preceding WQE addition operation has a clear address reference and guaranteeing that subsequent RDMA write operations can accurately locate the synchronization target.

[0075] In the above embodiments, a standardized association process between local and remote mapped memory regions is clearly defined, forming deep synergy with features such as dual-queue state awareness, dedicated queue pair configuration, and doorbell-triggered batch execution. Specifically, the exchange of registration information and the establishment of mirrored remote memory regions based on the RDMA protocol ensures accurate local awareness of remote thread states and accurate memory address association, resolving synchronization errors caused by address mismatches in traditional synchronization. Furthermore, the recording of the correspondence between local and remote memory regions provides a clear address configuration basis for "adding WQE before adding a new remote mapped memory region," making WQE configuration more accurate and efficient. These features, combined with hardware-driven batch execution and WQE reuse, further reduce the probability of errors in synchronization operations, improve the accuracy and stability of multi-node memory synchronization, and simplify the configuration logic of memory mapping between nodes, reducing software-level management complexity and overall improving the reliability and maintainability of memory synchronization in large-scale distributed systems.

[0076] In one embodiment, the hardware sequentially executes RDMA write operations corresponding to all work queue elements in the queue pair, synchronizing the updated local memory region data to all associated remote mapped memory regions, thus completing multi-node memory synchronization, including:

[0077] After receiving the preset trigger identifier of the doorbell address, the hardware reads the configuration information of each work queue element in the order in which the work queue elements are added to the queue pair, including the source address, destination address, local access key, remote access key and data length.

[0078] The hardware verifies access permissions to the local memory region based on the local access key and verifies access permissions to the remote mapped memory region based on the remote access key. Data transmission is initiated after the permission verification is successful.

[0079] Data is transferred directly between local memory and remote mapped memory via RDMA. After the RDMA write operation corresponding to all work queue elements is completed, the hardware sends a synchronization completion signal to the local node without sending an additional confirmation message to the remote node.

[0080] After receiving the preset trigger identifier of the doorbell address, the hardware first reads the complete configuration information of each WQE according to the order in which they were added to the queue. This information includes key details such as the source address, destination address, local / remote access key, and data length, ensuring the traceability of address and permission information for data transmission. Next, it verifies the access permissions of the local memory region and the remote mapped memory region using the local and remote access keys, respectively. Data transmission only begins after both ends have passed the permission verification, mitigating the risk of unauthorized access or data transmission errors at the permission level. Finally, it implements direct data transmission between the local and remote mapped memory regions based on RDMA technology, eliminating the need for operating system kernel intermediaries. Furthermore, after all RDMA write operations corresponding to all WQEs are completed, the hardware only sends a synchronization completion signal to the local node, without sending additional confirmation messages to the remote node, maximizing the simplification of the transmission process.

[0081] To achieve multi-node memory synchronization, a novel RDMA semantic is proposed: memSync(srcMem, dstMem1, ..., dstMem N), where srcMem represents the source memory region, and dstMem1, ..., dstMem N represent the set of remote target memory regions. Its operation mechanism includes:

[0082] 1) During the initialization phase, ensure that all target remote memory regions have corresponding WQEs added to QP and that they are fully integrated with the doorbell mechanism;

[0083] 2) During the data update phase, when data in the local memory area is updated, synchronization is triggered by knocking the doorbell, without needing to perform RDMA write on a target-by-target basis;

[0084] 3) During the hardware-triggered execution phase, the hardware sequentially executes all RDMA write completion synchronizations in the QP based on the notification.

[0085] The above embodiments clearly define the specific implementation methods for permission verification, ordered execution, and no additional confirmation messages, and deeply coordinate with features such as dual-queue state awareness, dedicated queue pair configuration, and doorbell triggering mechanism. Specifically, sequential reading of WQE configuration information ensures the orderliness of synchronization operations, avoiding state inconsistencies caused by out-of-order data transmission. Dual-end access key verification enhances the security of memory access, resolving permission risks associated with cross-node memory access in distributed systems. RDMA direct data transmission combined with the design of no remote confirmation messages eliminates the overhead of kernel relay and redundant feedback, further reducing synchronization latency. These features, in conjunction with designs such as WQE reuse, error-only CQE reporting, and pre-configured WQE, not only improve the security and orderliness of multi-node memory synchronization but also further compress synchronization time and reduce network and CPU resource consumption, significantly enhancing the efficiency, security, and stability of memory synchronization in large-scale distributed systems.

[0086] In one embodiment, the method further includes:

[0087] The system monitors the execution status of work queue elements in real time. When an error is detected in the execution of a work queue element, it receives the completed queue element reported by the hardware and initiates the error handling mechanism.

[0088] The above process primarily supplements the error monitoring and handling mechanism during multi-node memory synchronization, with a core focus on precise responses to abnormal scenarios. The system continuously and in real-time monitors the execution status of all work queue elements (WQEs) in the queue pair. Unlike the traditional full-state reporting mode, it triggers subsequent operations only for the abnormal scenario of "WQE execution error." Specifically, when an execution error is detected, the system receives a hardware-reported completion queue element (CQE), which contains key information such as the error type, the faulty WQE identifier, and the reason for failure. Based on this, the system immediately activates the preset error handling mechanism, rather than indiscriminately processing all WQE execution results, ensuring a targeted error response.

[0089] like Figure 6 As shown, when WQE N fails (i.e., synchronization between node 2's local subqueue and node N's remote subqueue fails), CQE N is sent. Upon receiving CQE, the system immediately initiates an error handling mechanism. If a hardware failure on a node during synchronization causes a WQE execution error, the system will attempt to re-execute the relevant RDMA write operation based on the error information in the CQE, or adjust the synchronization path according to a preset fault-tolerance strategy. Simultaneously, the system records detailed information about the error for subsequent fault analysis and system optimization.

[0090] In the above embodiments, real-time monitoring of the WQE execution status ensures that errors can be detected promptly, preventing the continuous spread of abnormal synchronization states. Receiving CQE and initiating the processing mechanism only when an error occurs continues the independent design philosophy of "reducing redundant feedback," avoiding ineffective state monitoring overhead in normal scenarios and reducing CPU resource consumption. Simultaneously, the error handling mechanism relies on accurate CQE information reported by the hardware, enabling rapid location of the erroneous WQE and its corresponding remote synchronization node. Combined with the design of WQE reuse and batch execution, retries or path adjustments can be efficiently initiated. This ensures the fault tolerance of multi-node memory synchronization without adding extra system burden due to error handling, further enhancing the stability and reliability of the synchronization process in large-scale distributed systems.

[0091] In one embodiment, the error handling mechanism's processing steps include:

[0092] After receiving a completion queue element, parse the error type, the identifier of the erroneous work queue element, and the information of the corresponding remote mapped memory region contained in the completion queue element.

[0093] In the event of a temporary communication exception, the corresponding error work queue element will be automatically re-executed.

[0094] If the error type is permission invalidation or invalid memory address, pause the synchronization of the corresponding remote mapped memory region, send an alarm message to the management user, and attach error details and handling suggestions;

[0095] After error handling is completed, the queue pairs resume normal operation.

[0096] After receiving a Complete Queue Element (CQE) reported by the hardware, the system first parses its core information, including the error type, the faulty work queue element (WQE) identifier, and the corresponding remote mapped memory region information, providing a precise basis for subsequent targeted processing. Then, it categorizes and processes errors according to type. For recoverable errors such as temporary communication anomalies, the system automatically triggers the re-execution of the corresponding faulty WQE, attempting repair without manual intervention. For unrecoverable errors such as invalid permissions or invalid memory addresses, the system suspends synchronization of the corresponding remote mapped memory region and sends an alarm message containing error details and processing suggestions to the management user, facilitating manual intervention for troubleshooting. Finally, after all error handling actions are completed, the system restores the normal operation of the queue pairs, ensuring that the synchronization process without errors is unaffected and guaranteeing the continuity of the overall synchronization mechanism.

[0097] like Figure 6As shown, when WQE N fails (i.e., synchronization between node 2's local subqueue and node N's remote subqueue fails), CQE N is sent. Upon receiving CQE, the system immediately initiates an error handling mechanism. If a hardware failure on a node during synchronization causes a WQE execution error, the system will attempt to re-execute the relevant RDMA write operation based on the error information in the CQE, or adjust the synchronization path according to a preset fault-tolerance strategy. Simultaneously, the system records detailed information about the error for subsequent fault analysis and system optimization.

[0098] In the above embodiments, by parsing CQE to locate error information, the problems of vague location and low investigation efficiency in traditional error handling are avoided. Automatic retrying of temporary anomalies reduces manual maintenance costs, while pausing synchronization and issuing alarms for permission / address errors avoids resource waste caused by invalid retries. After error handling is completed, the queue is restored to normal operation. Combined with WQE reuse and hardware batch execution design, this ensures accurate handling of individual error nodes without affecting the synchronization process of other remote nodes, further improving the fault tolerance and maintainability of multi-node memory synchronization. At the same time, the differentiated processing strategy takes into account both automated repair and manual intervention scenarios, reducing system maintenance complexity and promptly preventing the spread of serious errors, thus strengthening the stability and reliability of the synchronization mechanism in large-scale distributed systems.

[0099] In one embodiment, the registration process for the local memory region and the remote mapped memory region includes:

[0100] The local node registers the memory area where the storage thread is busy as a memory area, submits the virtual address, memory length and access permissions of the memory area to the network card, and the network card performs address translation and permission verification on the memory area, generates a local access key for local access and a remote access key for remote access, and returns them to the local node.

[0101] The local node synchronizes the remote access key, which allows remote nodes to access itself, and the basic information of the memory area to the associated remote node, so that the remote node can establish a mapped memory area; the basic information includes virtual address and memory length.

[0102] After receiving the remote access key and basic information about the memory area, the remote node registers the corresponding memory area as a local memory area and uses it as a remote mapped memory area.

[0103] The local node first registers the memory region where the storage thread is busy as a memory region (MR) under the RDMA specification, submitting core information such as the virtual address, memory length, and access permissions of the memory region to the network interface card (NIC). After receiving the information, the NIC performs address translation and permission verification, generating a local access key (Lkey) and a remote access key (Rkey) to distinguish between local and remote access, and returns them to the local node. This is the basis for enabling direct memory access. Subsequently, the local node synchronizes the remote access key and the basic information of the memory region (virtual address, memory length) to the associated remote node, providing a basis for the remote node to establish a mapped memory region. After receiving the information, the remote node registers the corresponding memory region as its own local memory region, and at the same time uses it as a remote mapped memory region pointing to the local node, completing the cross-node memory region association registration, ensuring that subsequent RDMA write operations can accurately and compliantly access the target memory region.

[0104] The above embodiments clearly define the specific processes for memory area registration, key generation, and cross-node information synchronization, deeply integrating with features such as exclusive dual-queue state awareness, dedicated queue pair configuration, and doorbell-triggered batch execution. Specifically, the standardized memory area registration and key generation via RDMA network cards ensure the compliance of local and remote memory access permissions, resolving the permission security issue of direct cross-node memory access in distributed systems. The local node synchronizes keys and basic memory information to the remote node, providing a unified and accurate basis for establishing the remote mapped memory area. Combined with the design of adding WQE at the front end, it provides a standardized data source for WQE source / destination addresses and access key configurations, avoiding synchronization failures caused by address or permission configuration errors. These features, along with hardware-driven batch execution, WQE reuse, and error-only CQE reporting, enhance the security and accuracy of multi-node memory synchronization, simplify the software logic of memory area registration, reduce the configuration complexity of cross-node memory mapping, and further improve the stability, security, and scalability of memory synchronization in large-scale distributed systems.

[0105] In one embodiment, the reuse and dynamic update process of work queue elements includes:

[0106] After each synchronization is complete, all work queue elements in the queue pair will retain their original configuration.

[0107] When adding a new remote mapped memory region, add a new work queue element to the queue pair. The new work queue element and the original work queue element are arranged in the order of addition and are triggered in sequence during execution.

[0108] If a remote mapped memory region is disconnected, the corresponding work queue element is removed from the queue pair; dynamic updates to work queue elements do not affect the currently executing synchronization process.

[0109] When the state of a local memory region changes and needs to be synchronized to all remote mapped memory regions, simply triggering the doorbell (writing a specific value to the doorbell address) completes the synchronization. Upon receiving the notification, the hardware executes all RDMA write operations within the QP (Queued Memory Area) to achieve synchronization. This QP has special properties: after a WQE (Write-Ahead Queue) operation, it doesn't need to be refilled and remains in the QP for future executions; and it only reports a CQE (Completion Queue Element) when an error occurs. For example, in a distributed system for real-time data processing, nodes continuously update their local data. Whenever a data update is complete, simply triggering the doorbell operation quickly and efficiently synchronizes the updated data to all relevant remote nodes without complex software intervention or significant CPU resource consumption.

[0110] In the above embodiments, the specific rules for WQE dynamic updates are refined, deeply integrating with features such as independent dual-queue state awareness, dedicated queue pair configuration, and doorbell-triggered batch execution. WQE reuse completely eliminates the software overhead of repeatedly creating and configuring WQEs. Combined with dynamic management of adding / deleting WQEs, it adapts to dynamic changes in node connections in distributed systems, improving the scalability of the synchronization mechanism. WQEs are executed in the order they are added, ensuring the orderliness of multi-node synchronization and avoiding state inconsistencies caused by out-of-order data transmission. Dynamic updates do not affect the design of the current synchronization process. Combined with hardware-driven batch execution and a mechanism that only reports errors to CQE, it ensures the continuity of synchronization operations while reducing the software intervention costs caused by changes in node connections. These features, working in conjunction with the core independent design, further reduce CPU resource consumption and software complexity, improving the flexibility, orderliness, and stability of memory synchronization in large-scale distributed systems, and adapting to complex application scenarios with dynamic additions and removals of nodes.

[0111] In one embodiment, the preset trigger identifier is in the format of binary data of a preset length, including queue pair identifier, synchronization instruction code and check bit.

[0112] The preset trigger identifier is defined as binary data of a preset length, unlike simple, unformatted numerical values. It contains three core fields: a queue pair identifier to precisely point to the specific queue pair requiring synchronization, avoiding signal confusion between different nodes or queue pairs; a synchronization instruction code, the core instruction for hardware to identify "starting the RDMA write operation execution process," ensuring the trigger signal's function is clearly defined; and a checksum to verify whether data errors occurred during transmission or writing, preventing false triggering or failure due to identifier corruption. The entire format design revolves around the precise identification, clear function, and data verification of the trigger signal, providing a standardized basis for hardware to respond quickly and accurately to synchronization trigger requests.

[0113] In the above embodiments, the queue pair identifier field corresponds to the design of "each node's dedicated queue pair" and "unique doorbell address," further strengthening the precise binding between the trigger signal and the execution carrier, and completely eliminating synchronization errors caused by signal mismatch. The synchronization instruction code ensures that the hardware can quickly identify and execute synchronization instructions, reducing signal parsing latency. The check bit reduces the risk of false triggering caused by incorrect trigger identifier data, improving the reliability of the triggering process. These features, combined with the hardware-led synchronous execution, WQE reuse, and error-only CQE design in the independent system, not only improve the accuracy and security of the synchronization triggering process but also accelerate the hardware's parsing and response speed to trigger signals, further reducing the overall latency of the synchronization process and strengthening the stability and efficiency of the memory synchronization triggering mechanism in large-scale distributed systems.

[0114] In one embodiment, the method further includes:

[0115] During the synchronization process, the progress data of the hardware execution work queue elements are collected in real time, including the number of work queue elements that have been completed, the number of remaining work queue elements, and the synchronization time.

[0116] Once all work queue elements have been executed without errors, the application layer on the local node reports a successful synchronization, along with the synchronization completion time and the amount of data synchronized.

[0117] If some work queue elements fail to execute during the synchronization process, the synchronization is completed after error handling and successful re-execution.

[0118] When the state of a local memory region changes and needs to be synchronized to all remote mapped memory regions, simply press the doorbell (write a specific value to the doorbell address). Upon receiving the notification, the hardware will execute all RDMA writes in the QP to complete the synchronization. This QP has special properties: after WQE is executed, it does not need to be refilled and will remain in the QP for the next execution; and CQE is only reported when an error occurs.

[0119] In the above embodiments, real-time progress data collection makes the synchronization process monitorable, facilitating timely detection of issues such as execution stalls and abnormal time consumption by maintenance personnel, thus improving system observability. The precise feedback mechanism tailored to different scenarios, combined with the design of reporting only errors via CQE and differentiated error handling, ensures that the application layer obtains complete synchronization results while avoiding CPU and network overhead caused by redundant feedback. The rule that synchronization failures require error handling and retrying before feedback is provided strengthens the integrity of data synchronization. Combined with the design of WQE reuse and hardware batch execution, this further enhances the reliability of multi-node memory synchronization. These features collectively reduce the operational costs of distributed system memory synchronization, enhance the controllability of synchronization results, and adapt to the needs of large-scale distributed systems for monitorable and traceable synchronization processes.

[0120] In one embodiment, such as Figure 3 The image shows a multi-node memory synchronization method in a specific embodiment. It includes:

[0121] Automatic Memory Synchronization Mechanism: A special reliable connectionless operation is created, and a unique memory synchronization process is built upon it. When an additional remote mapped memory region is added to the local memory region, an RDMAwrite is placed in the corresponding QP to write the state of the local memory region to the remote mapped memory region. These two steps are completed before actual synchronization as preparation for the control plane. Each QP is configured with a doorbell address to notify the hardware to execute the newly added WQE. Specifically, during the node initialization process of a distributed storage system, the system creates a special QP for the multi-node memory synchronization mechanism. When an additional remote mapped memory region is added to the local memory region, an RDMAwrite is placed in this QP, and its attributes are configured according to rules. For example, ... Figure 4 As shown, when node 2 establishes memory synchronization connections with multiple other nodes, the WQE that synchronizes the busy level of node 2's local memory region to the RDMA write operation of the remote node is pre-configured and associated with the corresponding doorbell address.

[0122] Synchronization Triggering and Execution Process: When the state of the local memory region changes and needs to be synchronized to all remote mapped memory regions, simply triggering the doorbell (writing a specific value to the doorbell address) completes the synchronization. Upon receiving the notification, the hardware executes all RDMA write operations within the QP (Queue of Completed Memory) to achieve synchronization. This QP has special properties: after WQE (Write-by-Door) execution, it does not need to be refilled and remains in the QP for future execution; and it only reports CQE (Completion Queue Element) when an error occurs. For example, in a distributed system for real-time data processing, nodes continuously update their local data. Whenever a data update is complete, simply triggering the doorbell operation quickly and efficiently synchronizes the updated data to all relevant remote nodes without complex software intervention or significant CPU resource consumption.

[0123] The new RDMA semantics: memSync semantic definition and operation mechanism: A new abstract semantic, memSync(srcMem, dstMem 1, dstMem N), is introduced to facilitate convenient multi-node memory synchronization operations. Here, srcMem represents the thread busy state information of the source memory, and dstMem 1, ..., dstMem N represent the set of remote target memory regions, to which the thread busy state information needs to be synchronized. In terms of the operation mechanism, firstly, during the initialization phase, it is ensured that all target remote memory regions have corresponding WQEs added to the QP, and that the doorbell mechanism is fully integrated. Secondly, during the data update phase, when data in the local memory region is updated, it is not necessary to perform RDMA writes target by target; only doorbell calls are needed. Finally, during the hardware-triggered execution phase, the hardware executes all RDMA writes in the QP sequentially according to the notification, completing the synchronization.

[0124] Optimization Features: The automatic memory synchronization mechanism boasts several significant optimization features. Regarding connectionless multi-target support, it achieves reliable connectionless synchronization, eliminating the need to establish dedicated connection paths for each target, greatly reducing connection management overhead and complexity. In terms of performance improvement, reusing WQE reduces creation and maintenance overhead, and direct hardware data push minimizes software intervention, making the data synchronization process more efficient and faster. For example, in a distributed financial transaction processing system with extremely high data synchronization latency requirements, reusing WQE and direct hardware data push can significantly reduce data synchronization time and improve the system's transaction processing efficiency. Regarding fault tolerance feedback, CQE reports are generated only when WQE execution fails, thereby reducing resource overhead under normal conditions, avoiding unnecessary error reporting and processing procedures, and improving system stability and reliability.

[0125] 1. System initialization phase:

[0126] During system startup, in this embodiment of the invention, a doorbell address corresponding to a multi-node memory synchronization mechanism is created for each node. During the creation of the QP, its attributes are configured according to the requirements of this invention, including associating it with a unique doorbell address. For each node, all remote nodes that need to be synchronized with its memory are first identified, and a WQE corresponding to the RDMA write operation that synchronizes the local memory region to the remote target memory region is added to the QP.

[0127] like Figure 4 As shown, in a distributed system consisting of N nodes, during initialization, node 2 establishes connections with the other N-1 nodes and creates a QP (QP2) for implementing a multi-node memory synchronization mechanism. At this point, for each connection of node 2, the system determines, according to a pre-defined memory synchronization strategy, which local memory regions (local memory regions 1-3) of node 2 need to be synchronized to the corresponding remote nodes, and determines which remote nodes node 2 needs to synchronize with. For each synchronization from a local memory region to a remote mapped memory region, a corresponding RDMA write operation WQE is generated and added to the specific QP already created for memory synchronization. Simultaneously, the association between the doorbell address and the QP is set to ensure the smooth operation of subsequent synchronization operations.

[0128] 2. Data update and synchronization phase:

[0129] When the busy status of a node's local memory area is updated, the system initiates the synchronization process by triggering the doorbell mechanism of the corresponding QP.

[0130] like Figure 5 As shown, in the distributed system consisting of N nodes, node 2 updates its local busy state buffer, and the gray area changes from 2 to 1, indicating that memory region 2 is no longer busy, while 1 is busy. At this time, a specific value is written to its doorbell address according to the QP (QP2) corresponding to node 2's memory synchronization mechanism. Upon receiving the notification, the hardware immediately begins executing the WQE for all RDMA writes in that QP, quickly synchronizing the updated data to all relevant remote nodes. At this point, position 1 in all remote mapped memory regions connected to node 2 from node 1 to node N is updated to busy. In this process, since the WQE is pre-configured during initialization and will always remain in the QP corresponding to the memory synchronization mechanism, only doorbell synchronization needs to be performed. Therefore, complex software reconfiguration and numerous function calls are unnecessary, greatly improving synchronization efficiency and reducing the latency from data update to synchronization completion.

[0131] 3. Error Handling and Monitoring Phase

[0132] The system continuously monitors the execution of WQE in QP, and only generates a completion queue element (CQE) report when an execution error occurs.

[0133] like Figure 6 As shown, when WQE N fails (i.e., synchronization between node 2's local subqueue and node N's remote subqueue fails), CQE N is sent. Upon receiving CQE, the system immediately initiates an error handling mechanism. If a hardware failure on a node during synchronization causes a WQE execution error, the system will attempt to re-execute the relevant RDMA write operation based on the error information in the CQE, or adjust the synchronization path according to a preset fault-tolerance strategy. Simultaneously, the system records detailed information about the error for subsequent fault analysis and system optimization.

[0134] 4. RDMA Multi-Node Memory Synchronization Semantics

[0135] To achieve multi-node memory synchronization, a novel RDMA semantic is proposed: memSync(srcMem, dstMem1, ..., dstMem N), where srcMem represents the source memory region, and dstMem1, ..., dstMem N represent the set of remote target memory regions. Its operation mechanism includes:

[0136] 1) During the initialization phase, ensure that all target remote memory regions have corresponding WQEs added to QP and that they are fully integrated with the doorbell mechanism;

[0137] 2) During the data update phase, when data in the local memory area is updated, synchronization is triggered by knocking the doorbell, without needing to perform RDMA write on a target-by-target basis;

[0138] 3) During the hardware-triggered execution phase, the hardware sequentially executes all RDMA write completion synchronizations in the QP based on the notification.

[0139] 4) During the error handling and monitoring phase, the system will continuously monitor the execution status of WQE in QP. Once CQE is received, the system will immediately start the error handling mechanism.

[0140] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0141] Based on the same inventive concept, this application also provides a large model tool invocation apparatus for implementing the large model tool invocation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the large model tool invocation apparatus provided below can be found in the limitations of the large model tool invocation method described above, and will not be repeated here.

[0142] In one exemplary embodiment, such as Figure 7 As shown, a multi-node memory synchronization device is provided, including: a creation module 702, a writing module 704, and a triggering module 706, wherein:

[0143] Create module 702 to establish connections between nodes in the distributed system. Each connection is configured with a local sub-queue for recording the busy level of local threads, and a remote sub-queue obtained by mapping the remote local sub-queue and used to record the busy level of remote threads.

[0144] The creation module 702 is also used to create queue pairs for each node and associate a unique doorbell address with each queue pair. The queue pairs are configured to be retained in the queue pairs after the work queue elements are executed, and to report the completed queue elements to the completion queue when an execution error occurs.

[0145] The write module 704 is used to add a corresponding RDMA write operation work queue element to the queue pair before actual data synchronization when a remote mapped memory region is added to the local memory region of the node. The work queue element defines the thread busy state of the local memory region to be written to the newly added remote mapped memory region.

[0146] The trigger module 706 is used to write a preset trigger flag to the doorbell address associated with the queue pair when the thread busy state in the local memory area is updated, so as to trigger the hardware to execute the RDMA write operation corresponding to all work queue elements in the queue pair in sequence, and synchronize the updated local memory area data to all associated remote mapped memory areas to complete multi-node memory synchronization.

[0147] In one embodiment, a creation module 702 is used to create queue pairs for each node and associate a unique doorbell address with each queue pair, including:

[0148] During system initialization, each node is allocated a queue pair resource for multi-node memory synchronization;

[0149] Configure the communication mode of the queue pair as reliable connectionless mode, configure the retention policy of the work queue elements in the queue pair as not to delete after execution and to retain them for reuse in the next synchronization, and configure the reporting policy of the queue elements as to only report when the work queue element execution fails.

[0150] Assign a unique doorbell address to each queue pair and establish a mapping relationship between the doorbell address and the queue pair.

[0151] In one embodiment, the writing module 704 is used for:

[0152] If the hardware detects a change in the data in the doorbell address in real time, and the change is a preset trigger identifier, it determines that a synchronization request has been received. The hardware then directly reads all configured work queue elements from the queue pair and starts the RDMA write operation execution process.

[0153] In one embodiment, the writing module 704 is used for:

[0154] When a connection is established between a local node and a remote node, the registration information of the memory regions of both parties is exchanged through the RDMA protocol. The registration information includes virtual address, memory length and remote access key.

[0155] Based on the registration information of the received remote memory region, a remote mapping memory region is established on the local node. The remote mapping memory region is a mirror mapping of the local sub-queue of the remote node.

[0156] Record the correspondence between the local memory region and the remote mapped memory region, and configure the source address and destination address in the work queue element based on the correspondence, so as to synchronously add the corresponding work queue element to the queue pair.

[0157] In one embodiment, the trigger module 706 is configured to:

[0158] After receiving the preset trigger identifier of the doorbell address, the hardware reads the configuration information of each work queue element in the order in which the work queue elements are added to the queue pair, including the source address, destination address, local access key, remote access key and data length.

[0159] The hardware verifies access permissions to the local memory region based on the local access key and verifies access permissions to the remote mapped memory region based on the remote access key. Data transmission is initiated after the permission verification is successful.

[0160] Data is transferred directly between local memory and remote mapped memory via RDMA. After the RDMA write operation corresponding to all work queue elements is completed, the hardware sends a synchronization completion signal to the local node without sending an additional confirmation message to the remote node.

[0161] In one embodiment, the method further includes:

[0162] The system monitors the execution status of work queue elements in real time. When an error is detected in the execution of a work queue element, it receives the completed queue element reported by the hardware and initiates the error handling mechanism.

[0163] In one embodiment, the creation module 702 is further configured to:

[0164] After receiving a completion queue element, parse the error type, the identifier of the erroneous work queue element, and the information of the corresponding remote mapped memory region contained in the completion queue element.

[0165] In the event of a temporary communication exception, the corresponding error work queue element will be automatically re-executed.

[0166] If the error type is permission invalidation or invalid memory address, pause the synchronization of the corresponding remote mapped memory region, send an alarm message to the management user, and attach error details and handling suggestions;

[0167] After error handling is completed, the queue pairs resume normal operation.

[0168] In one embodiment, the writing module 704 is further configured to:

[0169] The local node registers the memory area where the storage thread is busy as a memory area, submits the virtual address, memory length and access permissions of the memory area to the network card, and the network card performs address translation and permission verification on the memory area, generates a local access key for local access and a remote access key for remote access, and returns them to the local node.

[0170] The local node synchronizes the remote access key, which allows remote nodes to access itself, and the basic information of the memory area to the associated remote node, so that the remote node can establish a mapped memory area; the basic information includes virtual address and memory length.

[0171] After receiving the remote access key and basic information about the memory area, the remote node registers the corresponding memory area as a local memory area and uses it as a remote mapped memory area.

[0172] In one embodiment, the writing module 704 is further configured to:

[0173] After each synchronization is complete, all work queue elements in the queue pair will retain their original configuration.

[0174] When adding a new remote mapped memory region, add a new work queue element to the queue pair. The new work queue element and the original work queue element are arranged in the order of addition and are triggered in sequence during execution.

[0175] If a remote mapped memory region is disconnected, the corresponding work queue element is removed from the queue pair; dynamic updates to work queue elements do not affect the currently executing synchronization process.

[0176] In one embodiment, the preset trigger identifier is in the format of binary data of a preset length, including queue pair identifier, synchronization instruction code and check bit.

[0177] In one embodiment, the method further includes:

[0178] During the synchronization process, the progress data of the hardware execution work queue elements are collected in real time, including the number of work queue elements that have been completed, the number of remaining work queue elements, and the synchronization time.

[0179] Once all work queue elements have been executed without errors, the application layer on the local node reports a successful synchronization, along with the synchronization completion time and the amount of data synchronized.

[0180] If some work queue elements fail to execute during the synchronization process, the synchronization is completed after error handling and successful re-execution.

[0181] Each module in the aforementioned multi-node memory synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0182] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores work queue data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-node memory synchronization method.

[0183] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0186] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0189] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0190] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for multi-node memory synchronization, the method comprising: The method comprises: establishing connections between nodes in a distributed system, each connection being configured with a local sub-queue for recording the busy degree of a local thread and a remote sub-queue obtained by mapping the local sub-queue of a remote end and used for recording the busy degree of a remote thread; creating a queue pair for each node and associating a unique doorbell address with the queue pair, the queue pair being configured to retain a work queue element after execution of the work queue element in the queue pair and to report a completion queue element to a completion queue in the case of an execution error; in the case of adding a remote mapping memory region to the local memory region of a node, adding a work queue element corresponding to an RDMA write operation to the queue pair before actual data synchronization, the work queue element defining writing of the thread busy state of the local memory region to the added remote mapping memory region; in the case of an update of the thread busy state of the local memory region, writing a preset trigger identifier to the doorbell address associated with the queue pair to trigger the hardware to sequentially execute the RDMA write operation corresponding to all work queue elements in the queue pair and synchronize the updated local memory region data to all associated remote mapping memory regions, thereby completing the multi-node memory synchronization.

2. The method of claim 1, wherein, The method comprises: allocating a queue pair resource for multi-node memory synchronization to each node at system initialization; configuring the communication mode of the queue pair as a reliable connectionless mode, configuring the retention strategy of the work queue element in the queue pair as not deleting after execution and retaining for reuse in the next synchronization, and configuring the reporting strategy of the completion queue element as reporting only in the case of an execution error of the work queue element; allocating a unique doorbell address to the queue pair and establishing a mapping relationship between the doorbell address and the queue pair.

3. The method of claim 1, wherein, The implementation process of the doorbell address triggered synchronization comprises: in the case of a change in the data in the doorbell address and the change being a preset trigger identifier, determining that a synchronization request is received, directly reading all configured work queue elements from the queue pair by the hardware, and starting an RDMA write operation execution process.

4. The method of claim 1, wherein, The association process between the local memory region and the remote mapping memory region comprises: exchanging registration information of the memory regions of both parties through an RDMA protocol when establishing a connection between a local node and a remote node, the registration information comprising a virtual address, a memory length, and a remote access key; based on the received registration information of the remote memory region, establishing a remote mapping memory region in the local node, the remote mapping memory region being a mirror mapping of the local sub-queue of the remote node; recording the correspondence between the local memory region and the remote mapping memory region and configuring the source address and the destination address in the work queue element based on the correspondence, so as to synchronize the corresponding work queue element to the queue pair.

5. The method of claim 1, wherein, The hardware sequentially executes the RDMA write operation corresponding to all work queue elements in the queue pair, synchronizes the updated local memory region data to all associated remote mapping memory regions, and completes the multi-node memory synchronization. After receiving the preset trigger identifier of the doorbell address, the hardware reads the configuration information of each work queue element in turn according to the adding order of the work queue elements in the queue pair, including the source address, the destination address, the local access key, the remote access key and the data length; The hardware verifies the access permission of the local memory region based on the local access key, and verifies the access permission of the remote mapped memory region based on the remote access key, and starts data transmission after the permission verification is passed; Data is transmitted directly between the local memory and the remote mapped memory through the RDMA mode, and the hardware feeds back a synchronization completion signal to the local node after the execution of all the RDMA write operations corresponding to the work queue elements is completed, without sending an additional confirmation message to the remote node.

6. The method of claim 1, wherein, The method further comprises: Real-time monitoring of the execution state of the work queue elements, and receiving the completed queue element reported by the hardware and starting the error handling mechanism when an error in the execution of the work queue elements is detected.

7. The method of claim 6, wherein, The processing process of the error handling mechanism comprises: After receiving the completed queue element, the error type, the identifier of the error work queue element and the information of the corresponding remote mapped memory region contained in the completed queue element are parsed; In the case of temporary communication exception, the corresponding error work queue element is automatically triggered for re-execution; If the error type is permission invalidation or memory address invalidation, the synchronization of the corresponding remote mapped memory region is suspended, and an alarm information is sent to the management user, with error details and processing suggestions attached; After the error handling is completed, the normal operation of the queue pair is resumed.

8. The method of claim 1, wherein, The registration process of the local memory region and the remote mapped memory region comprises: The local node registers the memory region storing the thread busy state as a memory zone, and submits the virtual address, the memory length and the access permission of the memory zone to the network card, so that the network card performs address conversion and permission verification on the memory zone, generates a local access key for local access and a remote access key for remote access, and returns them to the local node; The local node synchronizes the remote access key for the remote node to access itself and the basic information of the memory zone to the associated remote node, so that the remote node establishes a mapped memory region; the basic information includes the virtual address and the memory length; After receiving the remote access key and the basic information of the memory zone, the remote node registers the corresponding memory region as a local memory zone and as a remote mapped memory region.

9. The method of claim 1, wherein, The reuse and dynamic updating process of the work queue elements comprises: After each synchronization is completed, all the work queue elements in the queue pair are kept in the original configuration; When a remote mapped memory region is added, a new work queue element is added to the queue pair, and the new work queue element and the original work queue elements are arranged in the adding order and triggered in sequence during execution; If a remote mapped memory region is disconnected, the corresponding work queue element is deleted from the queue pair; the dynamic updating of the work queue elements does not affect the current synchronization process being executed.

10. The method of claim 1, wherein, The format of the preset trigger identifier is a preset length of binary data, including a queue pair identifier, a synchronization instruction code and a check bit.

11. The method of claim 1, wherein, The method further comprises: In the synchronization process, progress data of the hardware execution work queue elements is collected in real time, including the number of executed work queue elements, the number of remaining work queue elements, and the synchronization time consumption; In the case that all work queue elements are executed without errors, the application layer of the local node is fed back with the synchronization success, the synchronization completion time length, and the synchronization data volume; In the case that part of the work queue elements fail in the synchronization process, the synchronization completion is fed back after the error processing is completed and the execution is successful.

12. A multi-node memory synchronization apparatus, comprising: The apparatus comprises: A creating module configured to establish connections between nodes in a distributed system, each connection being configured with a local sub-queue for recording the busy degree of a local thread and a remote sub-queue obtained by mapping the local sub-queue of a remote end and used for recording the busy degree of a remote thread; The creating module is further configured to create a queue pair for each node and associate a unique doorbell address with the queue pair, the queue pair being configured to retain work queue elements after execution and report completion queue elements to a completion queue in the case of execution error; A writing module configured to add work queue elements of corresponding RDMA write operations in the queue pair before actual data synchronization in the case that a remote mapping memory area is added in a local memory area of a node, the work queue elements defining writing the busy state of the thread of the local memory area to the added remote mapping memory area; A triggering module configured to write a preset trigger identifier to the doorbell address associated with the queue pair in the case that the busy state of the thread of the local memory area is updated, so as to trigger the hardware to sequentially execute the RDMA write operations corresponding to all work queue elements in the queue pair and synchronize the updated local memory area data to all associated remote mapping memory areas, thus completing the multi-node memory synchronization.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 11.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

Images