A method, apparatus, and storage medium for inter-process data exchange using a circular array based on step sequence positioning.
By constructing a two-segment mapping circular array inter-process data exchange method under the lockstep iteration mechanism, the metadata maintenance and synchronization operations are eliminated, and efficient data transmission for cross-process communication is achieved. This solves the problems of redundant overhead and displacement semantic mismatch in the existing technology and improves data transmission efficiency.
Patent Information
- Application Number
- CN202611336617.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from redundant overhead, mismatched displacement semantics, lack of consistent circular views, and reliance on explicit shared metadata in cross-process and cross-node communication, leading to increased communication time. In particular, under the lockstep iteration mechanism, the zero-movement benefit of the circular array cannot be effectively utilized.
By constructing a step-sequence-based inter-process data exchange method for circular arrays, a lockstep iteration mechanism is used to eliminate the need for location and notification metadata. A two-segment mapping rollback view and address translation structure are adopted to achieve automatic rollback across loop boundaries, avoiding the maintenance and synchronization of pointers and metadata.
It achieves a physical lower bound that approximates a single data transfer at the intra-node and cross-node interconnection levels, reducing communication overhead, improving data transmission efficiency, and lowering synchronization waiting time and metadata maintenance costs.
Smart Images

Figure CN123114797A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of inter-process data exchange technology in parallel computing, and in particular to a method, apparatus and computer-readable storage medium for inter-process data exchange of circular arrays based on step sequence positioning. Background Technology
[0002] In parallel computing, an important class of applications proceeds iteratively: the entire process follows a globally consistent lockstep iteration, exchanging data segments of known position and length with a statically determined set of neighbors at each communication step. Typical numerical solution applications (such as partial differential equation solving, particle simulation, and weather models) belong to this category, and neighbor data exchange is their main source of communication overhead.
[0003] To reduce data movement, working arrays are often organized as "circular arrays": data is stored in an array of length N, the algorithm maintains the starting offset, logical elements are mapped to physical locations through modulo operations, and a rotation of the starting point completes the logical translation of the data, without any actual movement of physical memory. The cost is that at any given time, a logically continuous interval to be swapped will be cut into two segments by the loop boundary when crossing the end of the array. It should be noted that circular arrays are essentially a computation-side optimization technique, benefiting from zero data movement; however, when this array is also used for data exchange across isolated memory (between processes or nodes), the aforementioned segmentation cost shifts from the computation side to the communication side. Existing communication paths organize data using linear displacement domains or message buffers, which cannot directly handle circular data. The movement saved on the computation side is retrieved again at process boundaries through split access, temporary storage, and segmented copying. How to maintain the zero-movement benefit of circular arrays in a distributed environment with isolated memory and make them connect both computing and communication sides is a problem that has not been solved by existing technology—this problem only arises when both the conditions of "circular arrays" and "inter-process data exchange" are met simultaneously.
[0004] The existing communication path and its shortcomings are as follows.
[0005] Path 1 within the node: Bilateral message passing. Data is moved multiple times through the sending buffer, the runtime internal buffer, and the receiving buffer, with both processes participating in the process.
[0006] Path 2 within the node: Shared memory window with load / store access. Each process directly accesses neighboring window data using local instructions, but its displacement domain is a linear interval, and data segments crossing loop boundaries must be retrieved twice; furthermore, this path does not support window registration in existing memory, and the application must first copy its working array into the window temporary storage area.
[0007] Cross-node path: Remote memory access window plus remote read. Each process exposes memory as a window, retrieving data from neighboring windows via remote read. Access must be performed within the access epoch of the global set operation switch. This path adds three layers of cost to the circular data: First, the window displacement domain is a linear interval, and cross-boundary reads must be split into two remote reads; second, the epoch switch is a set operation for all processes, requiring two executions per communication step, and any jitter in any process is amplified by the global lock step into a wait for all processes; third, remote reads have synchronous blocking semantics, and multi-directional reads can only be serially transmitted. In internal testing with two nodes and eight processes, the above three factors combined made the communication segment time of this path 2.5 times that of the cross-node path of this invention, with synchronous transmission accounting for approximately 83%, global epochs accounting for approximately 16%, and rollback splitting accounting for approximately 3%. It is evident that the main cost is not the data transport itself, but rather the synchronization and metadata mechanisms surrounding data location and access order.
[0008] In addition, the existing technology also has the following specific drawbacks: First, the displacement semantics do not match the cyclic data access pattern. The displacement domain of existing windows is a linear interval, and the accessor must perform modulo operation or mask rollback on the offset at the software level and handle segmented access on both sides of the boundary point, introducing additional branches and copies.
[0009] Secondly, the intra-process two-segment virtual mapping technique has not been used for inter-process communication. Within a single process, mapping the same physical memory segment to two adjacent equal-length intervals in the virtual address space allows out-of-bounds access to the circular data structure to automatically wrap back to the beginning of the segment after address translation. The application of this technique is limited to the implementation of data structures within a single process, and there is no solution to elevate it to an inter-process communication mechanism and ensure that multiple processes establish a consistent wrap-back view of the same physical segment.
[0010] Third, the combination of two-segment mapping and circular buffering is limited to fixed uses. Mapping the virtual address after the end of the circular buffer to the physical page where the first page of the buffer is located allows the buffer to present a continuous view locally, avoiding range checks and eliminating the need for splitting accesses across the end of the buffer. However, the objects constructed in this way are all dedicated buffers with fixed locations, requiring the maintenance and synchronization of head and tail pointers, setting arrival judgment fields according to messages, and clearing read areas by the receiver. There is no solution for applying aliasing mapping to working data arrays that rotate with communication steps, enabling pull-based remote reading step-by-step data retrieval without any pointer metadata.
[0011] Fourth, existing ring buffer communication schemes rely on explicit metadata. Whether it's a lock-free ring buffer (slot status, read / write pointers, size variables), a shared memory ring queue, or a remote memory ring message buffer (head and tail pointers, bell field), all require storing and maintaining metadata in shared or remote memory. This necessitates multiple parties performing atomic or remote operations, creating additional shared state, contention points, and consistency overhead. In the aforementioned general message buffer system, pointer and notification metadata are crucial elements—communication arrives in units of messages of any time and length. Without head and tail pointers, the receiver cannot locate the data; without notification fields, it's impossible to determine data arrival. Omitting pointer and notification metadata renders the message buffer communication system inoperable. This system lacks the premise that "data location can be calculated from the step sequence," therefore, omitting metadata is not possible.
[0012] Fifth, the above schemes share an unexamined institutional premise. Whether it's an intra-node circular queue or a cross-node circular message buffer, both assume communication occurs in units of messages of arbitrary timing, length, and order. Under this premise, metadata such as read / write pointers, slot status, and bell fields become indispensable elements. However, in a lockstep iterative system, the position, length, and timing of data exchange in each step are statically determined by the system itself. The aforementioned metadata is, in principle, redundant—there is no global step order in a message passing system, and omitting pointer metadata means the receiver cannot locate the data. Therefore, existing technologies neither recognize the possibility of omitting metadata nor provide corresponding technical guidance.
[0013] In summary, existing technologies suffer from drawbacks in cross-process and cross-node communication of cyclic data, including redundant overhead, mismatched displacement semantics, lack of a consistent cyclic view, and reliance on explicit shared metadata. Under lockstep iterative mechanisms, the metadata maintenance overhead related to data location and access order constitutes the main body of communication time consumption, which cannot be omitted by existing solutions due to their message buffering mechanism. A new communication mechanism is urgently needed that utilizes the lockstep iterative mechanism itself to eliminate location and notification metadata, enabling inter-process communication to approach the physical lower bound of a single data transfer at both the intra-node and cross-node interconnection levels. Summary of the Invention
[0014] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method, apparatus, and storage medium for inter-process data exchange using a circular array based on step-sequence positioning. Its purpose is to eliminate the positioning and notification metadata and its maintenance overhead (pointer updates, state clearing, arrival notification, synchronization round trips, and waiting) necessary for message buffering systems in an iterative communication framework where working data is organized in a circular array with logical starting points and the array must exchange neighboring data across isolated memory. Furthermore, it enables access across circular boundaries to be automatically rolled back by the address translation structure. This solves the technical problems of redundant overhead in existing inter-process communication paths, the inability of linear displacement domains to directly express the data access pattern of circular shifts, and the reliance on explicit shared metadata in ring-buffered communication. It allows the zero-movement benefit of the circular array to extend from the computation side to the communication side, approaching the physical lower bound of a single data transfer at both intra-node and inter-node interconnection levels.
[0015] To achieve the above objectives, according to a first aspect of the present invention, a method for inter-process data exchange using a circular array based on step sequence positioning is provided. The method includes a construction phase and a communication phase executed cyclically according to communication steps. The construction phase includes: A1. Each process participating in the communication constructs its own circular window: A shared memory segment with a length of N (page-aligned positive integer) is created or specified as a physical backup for the working data array that rotates roles with each communication step; a continuous virtual interval of length 2N is reserved in the process's virtual address space; the shared memory segment is mapped to the first and second halves of the continuous virtual interval using fixed addresses and the same intra-segment offset, forming a double-segment mapping rollback view; and the rollback relationship of the rollback view is configured to the address translation structure used by the communication access path, so that when the accessing party accesses the circular window via the address translation structure, the access interval crossing the end of the shared memory segment automatically rolls back to the beginning of the shared memory segment; A2. Initialize the step sequence flag and the window start point of each process; wherein, the communication neighbor set of each process and the position and length of the communication step exchange segment are statically determined before running; The communication phase includes: A3. Each process moves its own window starting point according to a consistent lockstep rotation rule. Each process can deduce the displacement of the current step data in any neighboring window based solely on the difference between its own window starting point and the base address. No pointers or metadata are exchanged throughout the process. A4. After each process confirms that all neighboring processes have completed reading the data from the previous step of this process, it writes the data to be exchanged in this step into the loop window of this process and releases a monotonically increasing step ready flag with release semantics. A5. Each process must satisfy two sequential conditions to obtain semantic confirmation: the current process has completed reading the previous step's data from the corresponding neighbor, and the neighbor has published the current step's data; A6. Each process determines the pull offset locally based on the difference between the starting point of its own window and the base address, performs a single continuous access to the loop window of the neighboring process, and reads the data required for this step into the landing point of its own process at once. When crossing the end of the shared memory segment, it does not split or pack the data. A7. After the pull is completed, each process updates the corresponding read completion flag with release semantics. The read completion flag is used by the corresponding neighbor process to confirm in step A4 of the next communication step. The loop window does not store read / write pointers or slot status metadata.
[0016] Preferably, in step A1, the shared memory segment is a named shared memory segment with a unique name in the operating system's shared memory namespace. The unique name is generated by the process identifier in the communication topology according to preset rules, and the communication parties can deduce the name of the other party's shared memory segment without negotiation. The creation adopts an exclusive creation flag, and the creator has ownership of the shared memory segment and is responsible for its reclamation. If a residual segment with the same name is found during creation, it is cleaned up and then rebuilt. The reserved contiguous virtual interval is achieved by requesting anonymous private mapping in protected state, and the fixed address mapping is achieved by superimposing a shared mapping on the reserved interval; In step A2, the initialized step ready flag and read complete flag are monotonically increasing step counters. The step number does not wrap around and reuse, thus avoiding the ABA problem (a type of concurrency error in lock-free concurrent operations where the value of a shared variable goes through the repetition of A→B→A, and the observer comparing values cannot detect the intermediate changes and misjudges that the state has not changed).
[0017] Preferably, for neighboring processes located on the same computing node: In step A1, the address translation structure is the processor page table of each participating process; the second process opens the shared memory segment of the first process with the unique name, and establishes the same double-segment mapping in the virtual address space of the process with the same segment length and the same segment boundary alignment relationship, so that each process has a logically equivalent consistent rollback view of the same shared memory segment. Each process also establishes the same two-segment mapping for the shared memory segment of the neighboring processes in each communication direction. In the case of bidirectional communication, a total of six virtual mappings are formed in each communication direction, and all mappings share physical memory and do not occupy additional physical memory. After step A2 and before entering the communication phase, each process exchanges and verifies the view fingerprint. The view fingerprint includes at least the segment length N, page alignment parameters, and number of mapped segments. If the verification is inconsistent, an error is reported and the mapping is reconstructed according to the other party's parameters.
[0018] Preferably, for neighboring processes located on other computing nodes: In step A1, the address translation structure is the address translation table of the network card that supports remote direct memory access; the first process registers the address translation relationship covering the continuous virtual interval as a whole with the network card, so that the address translation table of the network card inherits the aliasing relationship of the two-segment mapping page by page - the virtual pages corresponding to the first half and the second half point to the same set of physical page frames; thereafter, when the network card receives a remote access request falling in the second half, the address translation result automatically rolls back to the beginning of the shared memory segment; After registration is completed, each process will distribute access credentials to all participating processes through a one-time collection exchange. The access credentials include at least the virtual base address of the registration range and the remote access key.
[0019] Preferably, the shared memory segment carries a working data array that rotates with the communication step, the starting position of the rollback view shifts as a whole with the communication step, and the rollback point moves with the communication step; the working data array satisfies the dual criteria of a circular array: the structural criterion is that the working data array is carried by a single physical segment, and the second half of the virtual interval is a circular continuation of the first half; the semantic criterion is that when the fixed-length data retrieval window crosses the end of the shared memory segment, the two sides of the rollback point are the tail and head of the same logical array at the same time step, which are logically continuous; In step A3, the rotation operation translates the starting point of the window by a specified step size and rolls it back to the first half of the range. The rotation is a local operation of this process, which does not modify shared data and does not require inter-process negotiation.
[0020] Preferably, in step A2, the step-ready flag and the read-complete flag are arranged in the shared memory flag area, aligned with the cache line boundary, and read and write using lock-free atomic operations; When each process in step A6 is a multi-threaded fetch, the atomic increment of the last to the completed thread forms a release sequence, and then the read completion flag is updated; the single continuous access is a load / store instruction sequence. On platforms that use non-temporary storage to write back data, the deployment process performs a full barrier before the deployment step is ready flag to ensure that the non-temporary storage is globally visible.
[0021] Preferably, for neighboring processes located on other computing nodes: In step A4, the publishing process writes the step number to the persistent 8-byte mailbox slot of the pull process using a one-sided remote write; in step A5, the pull process polls the mailbox slot locally. In step A6, the single continuous access is a single remote read, which is automatically rolled back by the address translation table of the peer network card when it crosses the end of the shared memory segment; if the remote read operation fails, the process terminates immediately to prevent residual old data from participating in subsequent calculations. In step A7, the read completion flag is used to announce the progress of the unilateral remote write-back; the mailbox slots are persistently reused and determined by the monotonically increasing step number, without the need to be cleared.
[0022] Preferably, in step A6, the pull interval of a single continuous access is divided into several sub-intervals according to the number of participating threads, and multiple threads pull in parallel, with each thread bound to a designated physical core for execution; for cross-node directions, multiple remote reads are asynchronously transmitted and concurrently flown by multiple communication threads in their respective independent network card contexts; after all sub-intervals are completed and confirmed by the rendezvous point, step A7 is executed; communication threads and computing threads are allocated computing resources according to a preset ratio, and the pull operation and local computing overlap in time. When multiple communication neighbors of the same process are located at different interconnection layers, communication is automatically performed by selecting intra-node paths or cross-node paths in each direction according to the neighbor affiliation. The two paths share the same lockstep rotation state and step sequence system, and the path selection is transparent to upper-layer applications.
[0023] According to a second aspect of the present invention, a device for inter-process data exchange of a circular array based on step sequence positioning is provided, employing the above method, comprising: The construction module is used to construct the loop window of this process: create or specify a shared memory segment with a segment length of a page-aligned positive integer N as a physical backup for the working data array that rotates roles with the communication step; reserve a continuous virtual interval of length 2N in the virtual address space of this process; map the shared memory segment to the first half and the second half of the continuous virtual interval respectively with fixed addresses and the same intra-segment offset, forming a double-segment mapping rollback view; and configure the rollback relationship of the rollback view to the address translation structure used by the communication access path. The rotation module is used to shift the starting point of the window of the current process according to a consistent lockstep rotation rule at each communication step; The handshake module is used to release a monotonically increasing step-ready flag to release semantic release, to obtain semantic check step-read conditions, and to release a semantic update read-complete flag. The pull module is used to determine the pull displacement locally based on the difference between the starting point and the base address of the current process window, perform a single continuous access to the loop window of the neighboring process, and read the data required for this step into the current process's landing point at once. When crossing the end of the shared memory segment, it does not split or pack the data. The routing module is used to select paths between intra-node paths and cross-node paths in turn according to the interconnection level of communication neighbors. The loop window does not store read / write pointers or slot status metadata.
[0024] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the inter-process data exchange method for a circular array based on step sequence positioning as described in the first aspect.
[0025] In summary, compared with the prior art, the inter-process data exchange method, apparatus, and storage medium based on step-sequence positioning of circular arrays provided by the present invention have the following beneficial effects: 1. Zero metadata inter-addressing: Each process can deduce the displacement of the current step data in any neighboring window simply by the difference between the starting point and the base address of its own window. No read / write pointers or slot status metadata are stored in the circular window, eliminating the overhead of metadata maintenance, contention and consistency. In internal testing, the main thread communication wait time was reduced to about 1 / 12 of the baseline of the classic remote memory access window. 2. Data transport approaching the physical lower bound: Each communication step in each neighbor direction involves only a single continuous access, without splitting or packing when crossing the end of a shared memory segment, eliminating redundancy in buffer read / write, packing / unpacking copying, and runtime library advancement; internal tests show that intra-node path communication overhead is reduced by 23%–31% compared to shared memory window plus load / store access path; cross-node path, under a dual-node eight-process configuration, has a total step time that is reduced by approximately 58% compared to the classic remote memory access window baseline (the baseline total step time is approximately 2.4 times that of this invention). 3. Zero software overhead for rollback: The logical offset of the loop window is modulo the segment length N, and is automatically completed during address translation by the address translation structure (processor page table within the node, network card address translation table across nodes). Accessing the rollback does not require software branching, masking operations, segment copying, or remote reading and splitting. 4. Synchronous operation without global set operation: Neighbor handshake is completed by a monotonically increasing step ready flag and read complete flag, replacing the epoch switch of the entire process, thus avoiding the amplification of progress jitter caused by global step locking; step numbers are not wrapped around and reused, thus avoiding the ABA problem in terms of mechanism. 5. Uniformity and universality: Intra-node paths and cross-node paths share the same set of loop window semantics and step order system, transparently splitting traffic in each direction according to neighbor affiliation, without being bound to any specific application, and can be progressively integrated with existing parallel programming models. Attached Figure Description
[0026] Figure 1 This is a schematic diagram of the cross-process consistent rollback mapping address space provided by the present invention; Figure 2 This is a schematic diagram of the timing of the synchronous rotation window start point shift provided by the present invention; Figure 3 A schematic diagram of the atomic flag bidirectional handshake step state machine (path within a node) provided by the present invention. Figure 4 A flowchart illustrating the entire lifecycle of a loop window, from creation and connection to communication and dismantling, provided for this invention. Figure 5 This is a structural block diagram of the loop window communication device provided by the present invention; Figure 6 This is a schematic diagram illustrating the address translation method in the cross-node embodiment of the present invention, where aliasing registration enables the network interface card address translation table to inherit the rollback relationship. Figure 7 This is a timing diagram of mailbox handshake within a communication step in a cross-node embodiment provided by the present invention; Figure 8 This is a schematic diagram of the overall architecture for unifying intra-node paths and cross-node paths under the semantics of a loop window, as provided by the present invention. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0028] In this invention, "virtual address" refers to the address in the process address space; "physical address" refers to the address of the actual storage medium; "page table" refers to the data structure used by the processor's memory management unit for address translation; "address translation table" refers to the data structure used by the network interface card (NIC) to translate virtual addresses into physical addresses and verify access permissions; "registration" refers to the operation of the NIC registering a memory region and obtaining access credentials; "aliasing registration" refers to the same physical memory region being registered as accessible from two different virtual addresses, so that the NIC's address translation table gives the same physical translation result for the two virtual addresses; "circular array" refers to a logical array constructed from a single segment of physical memory through double-segment virtual mapping, where any interval of a certain length can be covered by a continuous virtual address; "step order" refers to the step count of a process advancing synchronously in rounds; "window start point" refers to the virtual address corresponding to the logical start point of the circular array, and its offset relative to the base address is called the rotation amount; "data publisher" refers to the process that writes data and issues a step-ready flag; "data consumer" (puller) refers to the process that confirms the step-order conditions and retrieves the data.
[0029] The inter-process data exchange method for circular arrays based on step sequence positioning provided in this invention is generally divided into a construction phase and a communication phase. Please refer to... Figure 4 This describes the entire lifecycle of a loop window, from creation, connection, communication to demolition; please refer to [link / reference]. Figure 8The overall architecture unifies intra-node paths and cross-node paths under the semantics of a circular window: modular arithmetic displacement domain, lockstep rotation, step-by-step handshake, and single continuous pull constitute a unified circular window semantic layer. Intra-node implementation paths (named shared memory, consistent two-segment view, atomic flag page, load / store access) and cross-node implementation paths (aliased registration, credential distribution, mailbox handshake, single remote read) implement this semantic layer in parallel. The rollback is uniformly absorbed by the address translation hardware, and the path selection is transparent to the upper-layer application.
[0030] This embodiment provides a method for inter-process data exchange using a circular array based on step-order positioning, including a construction phase and a communication phase executed cyclically according to communication steps, specifically including: Please see Figure 1 , Figure 4 and Figure 6 The construction phase includes: A1. Each process participating in the communication constructs its own circular window: A shared memory segment with a length of N (page-aligned positive integer) is created or specified as a physical backup for the working data array that rotates roles with each communication step; a continuous virtual interval of length 2N is reserved in the process's virtual address space; the shared memory segment is mapped to the first and second halves of the continuous virtual interval using fixed addresses and the same intra-segment offset, forming a double-segment mapping rollback view; and the rollback relationship of the rollback view is configured to the address translation structure used by the communication access path, so that when the accessing party accesses the circular window via the address translation structure, the access interval crossing the end of the shared memory segment automatically rolls back to the beginning of the shared memory segment; Specifically, it includes the following steps: S101. Creating a Named Shared Memory Segment: The first process creates a shared memory segment with an agreed-upon name in the operating system's shared memory namespace. This agreed-upon name is generated by the process identifiers in the communication topology according to preset rules, allowing each communicating party to deduce the segment name from the other without additional negotiation. An exclusive creation flag is used during creation. If a segment with the same name already exists (e.g., remnants from a previous run's abnormal exit), the creator performs cleanup before rebuilding. The creator sets the segment size, setting the segment length to a page-aligned positive integer N bytes; N also serves as the modulus of the window.
[0031] S102. Reserve a virtual interval of twice the length: The first process requests a continuous virtual interval of length 2N in its own virtual address space, reserving it without granting read or write permissions (protected reservation). This step ensures that the subsequent two mapped segments obtain virtual spaces with adjacent addresses and no overlap.
[0032] S103. Establish a two-segment overlay mapping: such as Figure 1As shown, the first process maps the shared memory segment to the first half [base, base+N) and the second half [base+N, base+2N) of the reserved interval using fixed addresses. Both mappings are shared mappings and map to the same physical segment and the same offset starting point. The resulting rollback view has the following properties: for any virtual address base+d (0≤d<2N), its access, after hardware address translation, falls to the d mod N byte of the physical segment; the window logical offset is automatically completed by address translation modulo the segment length N.
[0033] S104, (Intra-node path) Establish a cross-process consistent view: e.g. Figure 1 As shown, the second process (and each communicating neighbor process) opens the shared memory segment with the same name and repeats S102 and S103 in its own virtual address space, establishing a dual-segment mapping with the same segment length N and the same segment boundary alignment. The virtual base addresses of each process are allowed to be different, but the rollback relationship is logically equivalent in all processes. Each process establishes a dual-segment mapping for each neighbor segment in each communication direction; taking bidirectional communication as an example, a total of six segment mappings are formed in each direction (two segments for the current process and two segments for each of the two neighboring processes), and all mappings share the same physical memory, without occupying any additional physical pages.
[0034] S105, View Fingerprint Verification: During the connection establishment phase, each process exchanges view fingerprints (including at least the segment length N, page alignment parameters, and number of mapped segments). Only after the fingerprints match can the communication phase begin. If the fingerprints do not match, an error is reported and the mapping is reconstructed according to the other party's parameters to prevent silent data misalignment due to parameter mismatch.
[0035] S106, (Cross-node path) aliased registration: such as Figure 6 As shown, the first process registers the address translation relationship covering the 2N consecutive virtual intervals to the local network card (NIC) that supports Remote Direct Memory Access (RDMA). During registration, the NIC driver reads the process page table page by page and fills the NIC address translation table. Since the page table entries in the latter half point to the same set of physical page frames as the former half, the NIC address translation table inherits the aliasing relationship of the two-segment mapping page by page. Subsequently, when the NIC receives a remote access request falling in the latter half, the address translation result automatically rolls back to the beginning of the physical segment, and the initiator does not need any rollback judgment. After registration, a remote access key is obtained.
[0036] S107, (Cross-node path) Credential distribution: Each process distributes its access credentials (which at least include the virtual base address of the registered range and the remote access key) for each loop window of the process, as well as the address of the mailbox slot of the process, to all participating processes in a one-time set exchange; each process saves the credential table of all neighbors.
[0037] A2. Initialize the step sequence flag and the window start point of each process; wherein, the communication neighbor set of each process and the position and length of the communication step exchange segment are statically determined before running; Specifically, it includes the following steps: S108. Initialize step sequence flags: Arrange atomic step sequence flag groups aligned with cache line boundaries in the shared memory flag area of each process (or in a separate named shared memory area), including at least: the publish flag ready_step, and the read completion flags read_done from each direction; and arrange persistent 8-byte mailbox slots across node directions. Flags are initialized in-place and their lock-free attributes are verified; all flags are initialized to the start step value.
[0038] S109. Initialize Window Start Point: Each process initializes the window start point (start) to base, and the rotation (rot) is equal to start. With base=0, the system enters the communication ready state.
[0039] The pseudocode for the construction phase is shown in Algorithms 1 to 4, as shown in Tables 1-4 below.
[0040]
[0041]
[0042]
[0043]
[0044] The communication phase includes: A3. Each process moves its own window starting point according to a consistent lockstep rotation rule. Each process can deduce the displacement of the current step data in any neighboring window based solely on the difference between its own window starting point and the base address. No pointers or metadata are exchanged throughout the process. A4. After each process confirms that all neighboring processes have completed reading the data from the previous step of this process, it writes the data to be exchanged in this step into the loop window of this process and releases a monotonically increasing step ready flag with release semantics. A5. Each process must satisfy two sequential conditions to obtain semantic confirmation: the current process has completed reading the previous step's data from the corresponding neighbor, and the neighbor has published the current step's data; A6. Each process determines the pull offset locally based on the difference between the starting point of its own window and the base address, performs a single continuous access to the loop window of the neighboring process, and reads the data required for this step into the landing point of its own process at once. When crossing the end of the shared memory segment, it does not split or pack the data. A7. After the pull is completed, each process updates the corresponding read completion flag with release semantics. The read completion flag is used by the corresponding neighbor process to confirm in step A4 of the next communication step. The loop window does not store read / write pointers or slot status metadata.
[0045] For a detailed breakdown of the communication phase, please refer to [link / reference]. Figure 2 , Figure 3 , Figure 7 and Figure 8 The communication method mainly includes the following steps in each communication step: S201, Lockstep rotation: such as Figure 2 As shown, during the communication step, each participating process rotates its own window according to a consistent rule (calling Algorithm 3). Since all processes execute the same rotation rule, any process can deduce the displacement of the data in the current step in any neighboring window simply by using its own rotation, without needing to exchange pointers or any metadata. The displacement field of the cyclic window in this invention is a modulo arithmetic displacement field modulo N, and the rollback is completed by hardware address translation; while the linear displacement field of the existing communication window requires modulo / masking operations and segmented access at the software level.
[0046] S202, Publishing Data: The publishing process writes the data to be exchanged in this step into the agreed area in the window of this process, with start as the reference; after writing, the publishing flag ready_step is updated to the current step number for neighbors within the node using release semantics; for neighbors across nodes, the step number is written to the mailbox slot of the puller using a one-sided remote write (8 bytes of remote write at a time, with zero participation from the peer processor).
[0047] S203, Waiting for Step Sequence Conditions: The pull process checks two step sequence conditions for each neighbor direction: (i) the current process has completed reading the previous step's data from that neighbor; (ii) the neighbor has published the current step's data. Intra-node directions retrieve semantic read shared memory atomic flags; cross-node directions retrieve semantic local polling mailbox slots. If the conditions are not met, the process spins and waits or yields the processor.
[0048] S204, Single Continuous Fetch: After the conditions are met, the fetch process calculates the fetch displacement based on rot and initiates a single continuous access to the neighbor window, reading the data required for this step into the process's endpoint at once: Within a node, the direction is a load / store instruction sequence (the source end is within the neighbor consistency view, and the destination end is within the process's rollback view, both virtually continuous); across nodes, it is a single remote read (source address = neighbor registration base address + rot + target segment offset; when crossing a loop boundary, the peer network card address is translated automatically and the data rollback is not split). For each neighbor direction, this data transfer occurs only once per step.
[0049] S205, Declaring Read Completion: After the pull is completed, the intra-node direction first performs an atomic increment on the pull completion count (get-release semantics), forming a release sequence with the corresponding read on the neighbor side, and then updates the corresponding read completion flag with release semantics; the cross-node direction sends a read completion notification to the publisher in one 8-byte mailbox write. Before overwriting the corresponding area in the next communication step, the publisher must confirm that the read completion flag in each direction is not less than that in the current step. Figure 3 The step sequence state machine of the bidirectional handshake of the directional atomic flags in the node is shown: the puller goes through the states of "waiting condition (i) - waiting condition (ii) - single pull - update read completion flag" in sequence. The step counter is monotonically increasing and the mechanism prevents ABA. The flag group is aligned according to the cache line to avoid false sharing.
[0050] S206, Multi-threaded Resource Allocation and Computation Overlap: When a single thread's bandwidth is insufficient, the fetch interval is evenly divided according to the number of threads, and multiple threads bound to a specified physical core fetch resources in parallel. Multiple remote reads across nodes are asynchronously transmitted and concurrently flown by multiple communication threads in their respective independent network interface card contexts. After all threads have completed, the rendezvous point confirms the completion, and then S205 is executed. Communication threads and computation threads are allocated resources according to a preset ratio, and the fetching and local computation pipelines overlap.
[0051] S207, Hybrid Flow: When two neighbors of the same process are located at different interconnection levels, the node instance within or across nodes of S202–S205 is automatically selected in each direction according to the neighbor's affiliation; the two sub-paths share the same lockstep rotation state (S201) and step sequence system, and the path selection is transparent to the upper layer application. Figure 7 The complete timing of the cross-node direction mailbox handshake within a communication step is shown: After the publisher writes data, it writes the step sequence number to the mailbox slot of the puller in an 8-byte one-sided remote write. After the puller polls locally and the step sequence condition is met, it initiates a single remote read (the peer network card address translation table absorbs and rolls back, without splitting). After the data is returned, it writes back the read result in an 8-byte mailbox to indicate that the read is complete. There are no global set operations throughout the process, and the slots are persistently reused and do not need to be cleared.
[0052] The pseudocode for the communication phase is shown in Algorithms 5 to 8, as shown in Table 5-8 below.
[0053]
[0054]
[0055]
[0056]
[0057] Please see Figure 5 This invention also provides a step-sequence-based inter-process data exchange device for circular arrays, deployed in each participating process, including: The construction module is used to construct the loop window of this process: create or specify a shared memory segment with a segment length of a page-aligned positive integer N as a physical backup for the working data array that rotates roles with the communication step; reserve a continuous virtual interval of length 2N in the virtual address space of this process; map the shared memory segment to the first half and the second half of the continuous virtual interval respectively with fixed addresses and the same intra-segment offset, forming a double-segment mapping rollback view; and configure the rollback relationship of the rollback view to the address translation structure used by the communication access path. The rotation module is used to shift the starting point of the window of the current process according to a consistent lockstep rotation rule at each communication step; The handshake module is used to release a monotonically increasing step-ready flag to release semantic release, to obtain semantic check step-read conditions, and to release a semantic update read-complete flag. The pull module is used to determine the pull displacement locally based on the difference between the starting point and the base address of the current process window, perform a single continuous access to the loop window of the neighboring process, and read the data required for this step into the current process's landing point at once. When crossing the end of the shared memory segment, it does not split or pack the data. The routing module is used to select paths between intra-node paths and cross-node paths in turn according to the interconnection level of communication neighbors. The loop window does not store read / write pointers or slot status metadata.
[0058] In addition, the present invention also provides the following embodiments.
[0059] Example 1: Minimal Two-Process Ping-Pong Communication Example. Two processes, P0 and P1, reside on the same node. P0 creates the segment " / cw_p0" according to steps S101–S109, establishes a two-segment mapping, and initializes flags; P1 similarly creates " / cw_p1" and, according to step S104, opens P0's segment with the same name to establish a consistent view. P0 also performs the same processing on " / cw_p1". Both parties execute Algorithm 5 in each communication step. This example verifies with minimal configuration: a cross-process consistent view can be established, zero-metadata location is correct, and the handshake protocol does not deadlock or overwrite. See also... Figures 1-4 .
[0060] Example 2: Complete Implementation of a Many-Core Platform with Multi-Process and Multi-Thread (Intra-Node Path). On a single-node 304-core many-core chip, processes are arranged in a nearest-neighbor topology, with each process having two communication directions (up and down). Each process establishes a dual-segment mapping for its own segment and two neighboring segments, for a total of six segment mappings per direction. The total physical memory is the same as for a single segment. Communication is executed according to S201–S207, where S206 enables a ratio of 32 computing threads to 4 communication threads. The fetch interval is evenly distributed among threads, and threads are bound to physical cores. Flag groups are arranged aligned to 64-byte cache lines. On platforms using non-temporary storage for write-back data, the publisher executes a full barrier before updating ready_step. Internal testing shows that compared to shared memory windows plus load / store access paths, this embodiment reduces communication overhead by 23%–31%, and the overall update rate reaches 17.19 GLUPS (grid unit time step update rate). See also... Figures 1-3 .
[0061] Example 3: Complete Cross-Node Implementation (Cross-Node Path). Eight processes run on two nodes interconnected via a RoCE-enabled network, four per node. The processes are arranged in a one-dimensional chain-like nearest neighbor topology, with each process having both up and down directions. Each process holds several cyclic windows (each window segment is several megabytes long and page-aligned). After each process establishes a two-segment mapping for each window according to S101–S103, aliasing registration is performed on the entire 2N virtual interval of each window according to S106, and credential distribution is completed according to S107.
[0062] Each communication step: Each process first locks its rotation (S201); boundary data is written to the process window using vector instructions (the platform using non-temporary storage executes a full barrier before publication); the step sequence number is written to the puller's mailbox slot via a one-sided remote write for cross-node directions (S202); after the puller's local polling mailbox meets the step sequence conditions (S203), the displacement is derived using local rot, and a single remote read is initiated to the neighbor window—in this embodiment, each step reads a complete plane (approximately 256KB) in each direction, and the remote read is asynchronously transmitted and concurrently flown by four 64KB pipelines; after the read is completed, a mailbox write is sent back (S205). The reading across the loop boundary is not split throughout: internal special experiments verify the full range scan of the aliased registration area, the sweep across the rollback point, and the 2000 sets of random offset / length reads are all correct; the live protocol experiment runs 20,000 steps continuously with a rotation sequence across the rollback point almost every step, and there is zero mismatch between the non-temporary storage and ordinary storage in both rounds.
[0063] Internal testing comparing this implementation with the classic fence-type remote memory access window path (same physical data, address mathematical equivalence confirmed by three independent audits): the classic path's communication segment time is 2.5 times that of this embodiment, the main thread communication wait time is 12 times longer, and the total step time is approximately 2.4 times longer. This means the total step time of this embodiment is reduced by approximately 58% compared to the baseline. The baseline time is attributed item-by-item to approximately 83% for synchronous blocking launch, approximately 16% for global epoch synchronization, and approximately 3% for rollback splitting. See also... Figures 6-8 .
[0064] Example 4: Hybrid Mode Example. For applications spanning multiple computing nodes, intra-node neighbors communicate via intra-node paths (Algorithm 5), while cross-node neighbors communicate via cross-node paths (Algorithm 6), with S207 automatically routing communication based on neighbor affiliation in each direction. The upper-layer application code only needs to declare the node affiliation of neighbors during initialization; both sub-paths share the same lockstep rotation state and step number system, and the communication path selection is transparent to the application. This example verifies the composability of the two instantiation methods of this invention and its progressive integration capability with existing parallel programming models.
[0065] Example 5: An example of a lattice Boltzmann method based on the Periodic Shift algorithm. This type of method stores distribution function data in a single physical array, where the array carries the same logical field data at any given time step. No data is moved during the flow step; instead, the data role is rotated by periodically shifting the array's logical starting point using modulo operations—rotation is an inherent property of the algorithm's core, and a circular array is its natural data organization form. In the distributed implementation, processes are neighbors according to grid adjacency relationships. At each time step, the process exchanges the position and length of the boundary layer distribution function with its statically determined neighbors. When the data retrieval window crosses the physical end of the array, the data at the beginning of the same array is directly read through a two-segment mapping. The two sides of the rollback point are the tail and head of the same logical array, satisfying the dual criteria for circular arrays described in this invention. With this invention, cross-process / cross-node reading of boundary layer data is completed by a single continuous pull through the circular window, without splitting or copying when crossing the end of the array (see Examples 2 and 3 for specific values).
[0066] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for inter-process data exchange using a circular array based on step sequence positioning, characterized in that, The method includes a construction phase and a communication phase executed cyclically according to communication steps, wherein the construction phase includes: A1. Each process participating in the communication constructs its own circular window: A shared memory segment with a length of N (page-aligned positive integer) is created or specified as a physical backup for the working data array that rotates roles with each communication step; a continuous virtual interval of length 2N is reserved in the process's virtual address space; the shared memory segment is mapped to the first and second halves of the continuous virtual interval using fixed addresses and the same intra-segment offset, forming a double-segment mapping rollback view; and the rollback relationship of the rollback view is configured to the address translation structure used by the communication access path, so that when the accessing party accesses the circular window via the address translation structure, the access interval crossing the end of the shared memory segment automatically rolls back to the beginning of the shared memory segment; A2. Initialize the step sequence flag and the window start point of each process; wherein, the communication neighbor set of each process and the position and length of the communication step exchange segment are statically determined before running; The communication phase includes: A3. Each process moves its own window starting point according to a consistent lockstep rotation rule. Each process can deduce the displacement of the current step data in any neighboring window based solely on the difference between its own window starting point and the base address. No pointers or metadata are exchanged throughout the process. A4. After each process confirms that all neighboring processes have completed reading the data from the previous step of this process, it writes the data to be exchanged in this step into the loop window of this process and releases a monotonically increasing step ready flag with release semantics. A5. Each process must satisfy two sequential conditions to obtain semantic confirmation: the current process has completed reading the previous step's data from the corresponding neighbor, and the neighbor has published the data for this step. A6. Each process determines the pull offset locally based on the difference between the starting point of its own window and the base address, performs a single continuous access to the loop window of the neighboring process, and reads the data required for this step into the landing point of its own process at once. When crossing the end of the shared memory segment, it does not split or pack the data. A7. After the pull is completed, each process updates the corresponding read completion flag with release semantics. The read completion flag is used by the corresponding neighbor process to confirm in step A4 of the next communication step. The loop window does not store read / write pointers or slot status metadata.
2. The inter-process data exchange method for circular arrays based on step sequence positioning as described in claim 1, characterized in that: In step A1, the shared memory segment is a named shared memory segment with a unique name in the operating system's shared memory namespace. The unique name is generated by the process identifier in the communication topology according to preset rules. The communication parties can deduce the name of the other party's shared memory segment without negotiation. The creation adopts an exclusive creation flag. The creator has ownership of the shared memory segment and is responsible for its reclamation. If a residual segment with the same name is found during creation, it is cleaned up and then rebuilt. The reserved contiguous virtual interval is achieved by requesting anonymous private mapping in protected state, and the fixed address mapping is achieved by superimposing a shared mapping on the reserved interval; In step A2, the initialized step ready flag and read complete flag are monotonically increasing step counters, and the step number is not wrapped around and reused.
3. The inter-process data exchange method for circular arrays based on step sequence positioning as described in claim 2, characterized in that, For neighboring processes located on the same compute node: In step A1, the address translation structure is the processor page table of each participating process; the second process opens the shared memory segment of the first process with the unique name, and establishes the same double-segment mapping in the virtual address space of the process with the same segment length and the same segment boundary alignment relationship, so that each process has a logically equivalent consistent rollback view of the same shared memory segment. Each process also establishes the same two-segment mapping for the shared memory segment of the neighboring processes in each communication direction. In the case of bidirectional communication, a total of six virtual mappings are formed in each communication direction, and all mappings share physical memory and do not occupy additional physical memory. After step A2 and before entering the communication phase, each process exchanges and verifies the view fingerprint. The view fingerprint includes at least the segment length N, page alignment parameters, and number of mapped segments. If the verification is inconsistent, an error is reported and the mapping is reconstructed according to the other party's parameters.
4. The inter-process data exchange method for circular arrays based on step sequence positioning as described in any one of claims 1 to 3, characterized in that, For neighboring processes located on other computing nodes: In step A1, the address translation structure is the address translation table of the network card that supports remote direct memory access; the first process registers the address translation relationship covering the continuous virtual interval as a whole with the network card, so that the address translation table of the network card inherits the aliasing relationship of the two-segment mapping page by page - the virtual pages corresponding to the first half and the second half point to the same set of physical page frames; thereafter, when the network card receives a remote access request falling in the second half, the address translation result automatically rolls back to the beginning of the shared memory segment; After registration is completed, each process will distribute access credentials to all participating processes through a one-time collection exchange. The access credentials include at least the virtual base address of the registration range and the remote access key.
5. The inter-process data exchange method for circular arrays based on step sequence positioning as described in any one of claims 1 to 4, characterized in that: The shared memory segment carries the working data array that rotates with the communication step. The starting position of the rollback view shifts as a whole with the communication step, and the rollback point moves with the communication step. The working data array satisfies the dual criteria of a circular array: the structural criterion is that the working data array is carried by a single physical segment, and the virtual interval of the second half is a circular continuation of the first half; the semantic criterion is that when the fixed-length data retrieval window crosses the end of the shared memory segment, the two sides of the rollback point are the tail and head of the same logical array at the same time step, which are logically continuous. In step A3, the rotation operation translates the starting point of the window by a specified step size and rolls it back to the first half of the range. The rotation is a local operation of this process, which does not modify shared data and does not require inter-process negotiation.
6. The inter-process data exchange method for circular arrays based on step sequence positioning as described in any one of claims 1 to 5, characterized in that, For neighboring processes located on the same compute node: In step A2, the step-ready flag and the read-complete flag are arranged in the shared memory flag area, aligned with the cache line boundary, and read and write using lock-free atomic operations; When each process in step A6 is a multi-threaded fetch, the atomic increment of the last to the completed thread forms a release sequence, and then the read completion flag is updated; the single continuous access is a load / store instruction sequence. On platforms that use non-temporary storage to write back data, the deployment process performs a full barrier before the deployment step is ready flag to ensure that the non-temporary storage is globally visible.
7. The inter-process data exchange method for circular arrays based on step sequence positioning as described in any one of claims 1 to 5, characterized in that, For neighboring processes located on other computing nodes: In step A4, the publishing process writes the step number to the persistent 8-byte mailbox slot of the pull process using a one-sided remote write; in step A5, the pull process polls the mailbox slot locally. In step A6, the single continuous access is a single remote read, which is automatically rolled back by the address translation table of the peer network card when it crosses the end of the shared memory segment; if the remote read operation fails, the process terminates immediately to prevent residual old data from participating in subsequent calculations. In step A7, the read completion flag is used to announce the progress of the unilateral remote write-back; the mailbox slots are persistently reused and determined by the monotonically increasing step number, without needing to be cleared.
8. The inter-process data exchange method for circular arrays based on step sequence positioning as described in any one of claims 1 to 7, characterized in that: In step A6, the pull interval of a single continuous access is divided into several sub-intervals according to the number of participating threads, and multiple threads pull in parallel, with each thread bound to a specified physical core for execution; For cross-node directions, multiple remote reads are asynchronously transmitted and concurrently flown by multiple communication threads in their respective independent network interface card contexts; after all sub-intervals are completed and confirmed by the rendezvous point, step A7 is executed; communication threads and computing threads allocate computing resources according to a preset ratio, and the pull operation and local computing overlap in time pipeline; When multiple communication neighbors of the same process are located at different interconnection layers, communication is automatically performed by selecting intra-node paths or cross-node paths in each direction according to the neighbor affiliation. The two paths share the same lockstep rotation state and step sequence system, and the path selection is transparent to upper-layer applications.
9. A step-sequence-based inter-process data exchange device for a circular array, employing the method described in any one of claims 1 to 8, characterized in that, include: The construction module is used to construct the loop window of this process: create or specify a shared memory segment with a segment length of a page-aligned positive integer N as a physical backup for the working data array that rotates roles with the communication step; reserve a continuous virtual interval of length 2N in the virtual address space of this process; map the shared memory segment to the first half and the second half of the continuous virtual interval respectively with fixed addresses and the same intra-segment offset, forming a double-segment mapping rollback view; and configure the rollback relationship of the rollback view to the address translation structure used by the communication access path. The rotation module is used to shift the starting point of the window of the current process according to a consistent lockstep rotation rule at each communication step; The handshake module is used to release a monotonically increasing step-ready flag to release semantic release, to obtain semantic check step-read conditions, and to release a semantic update read-complete flag. The pull module is used to determine the pull displacement locally based on the difference between the starting point and the base address of the current process window, perform a single continuous access to the loop window of the neighboring process, and read the data required for this step into the current process's landing point at once. When crossing the end of the shared memory segment, it does not split or pack the data. The routing module is used to select paths between intra-node paths and cross-node paths in turn according to the interconnection level of communication neighbors. The loop window does not store read / write pointers or slot status metadata.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the inter-process data exchange method for a circular array based on step sequence positioning as described in any one of claims 1 to 8.
