A discrete data stream memory access optimization method based on sunway many-core processor

Abstract

The application relates to a discrete data flow memory access optimization method based on a Sunway many-core processor SW39000, and belongs to the technical field of electronic information, and comprises the following steps: step 1, mapping and rearranging discrete data in a main core; including: constructing a space grid mapping reference according to node vector information, establishing a linear mapping relationship of discrete data objects to a grid index, and performing binning and global rearrangement on the data objects according to the linear mapping relationship, so that data objects with similar spatial positions are stored in positions close to each other in a main memory; and step 2, block calculation and split aggregation transmission; including: executing data slicing and split aggregation DMA strategies on a slave core side, and combining data locality established in the main core to reconfigure inefficient random memory access into efficient flow data multiplexing. The application significantly improves the memory access locality of discrete data in a many-core environment, and solves the bandwidth bottleneck problem caused by large-scale random memory access.

Description

Technical Field

[0001] This invention belongs to the technical field of electronic information, specifically relating to an optimization method for discrete data streaming memory access based on Shenwei many-core processors. Background Technology

[0002] In recent years, the development of supercomputers has leaped from petascale (PetaFLOPS) to exascale (E-class). The development of domestically produced supercomputers not only demonstrates my country's independent innovation capabilities in chip design, interconnect networks, and system software, but also provides core computing power support for solving key scientific problems in strategic fields such as climate simulation, nuclear fusion research, and aerospace. This success is inseparable from the continuous iteration of domestically produced high-performance many-core processor architectures and independent breakthroughs in core technologies, with the Shenwei series processors being the most representative example.

[0003] The "Sunway Blue Light II" is a significant milestone in my country's high-performance computing systems. Its core computing power is provided by the SW39000 Shenwei many-core processor, which possesses completely independent intellectual property rights. As a typical domestically developed heterogeneous many-core architecture, the SW39000 adopts an on-chip integrated heterogeneous technology approach in its chip design, aiming to achieve high-throughput numerical computing capabilities through the combination of massively parallel computing units and explicit memory hierarchies. Its hardware architecture is as follows... Figure 1 As shown, the processor integrates 390 processing units in its computing core layout, logically divided into six symmetrical core groups (CGs). Each core group constitutes an independent NUMA (Non-Uniform Memory Access) node, containing one management processing element (MPE) responsible for task scheduling and global control, and an 8×8 high-density computing array consisting of 64 computing processing elements (CPEs). This collaborative mode of "master core control, slave core computation" is the basic form of the Shenwei architecture for handling large-scale parallel tasks.

[0004] In terms of the storage subsystem and memory access mechanism, the SW39000 adopts a software-controllable explicit storage hierarchy. Its key feature is the physical division of the storage space into two levels: the first level is a large-capacity global main memory (DDR4) shared among cores; the second level is a tightly coupled private local data storage space (LDM) located within each slave core. Unlike the cache in traditional architectures, which relies entirely on hardware automatic management, the LDM (256KB capacity) in the Shenwei architecture is primarily designed as a notepad-like memory, allowing the compiler to precisely control data residency and swapping through instructions.

[0005] To enable efficient data flow between heterogeneous storage tiers, the SW39000 provides two standard data transfer paths: one is discrete memory access (gld / gst), where the slave core can directly access the main memory space through global addresses without using LDM as a data cache; the other is direct memory access (DMA), where the slave core can initiate DMA operations to realize batch data transfer between LDM and main memory, supporting the transfer of continuous data blocks in burst mode.

[0006] In large-scale scientific computing fields such as plasma physics simulation, fluid dynamics, and molecular dynamics, programs typically need to track the trajectories of hundreds of millions of discrete computational elements in continuous space. When such applications run on the Shenwei many-core processor, they exhibit a coexistence of computationally intensive and discrete memory access: when the slave core array executes a high-order update algorithm for each computational element, it must frequently read global background field data near its current coordinates from main memory. This computing mode presents a significant challenge to the Shenwei architecture: because computational elements move randomly in space, while background field data is statically and linearly stored in memory, slave core access requests to main memory exhibit extreme disorder. Although the Shenwei processor provides two mechanisms—discrete memory access (gld / gst) and direct memory access (DMA)—both struggle with these applications: direct discrete memory access suffers from significant latency due to a lack of cache support; while DMA, although supporting efficient batch burst transfers, has strict requirements for data address continuity, making it difficult to adapt to discrete data objects.

[0007] The mismatch between the aforementioned discrete computing model and the continuous transfer mechanism manifests in three technical obstacles during actual program execution. First, the characteristics of discrete data objects often exhibit highly random distribution and dynamic changes. This disordered data distribution leads to a large number of memory access requests with discrete addresses, large jumps, and irregularities when the slave core retrieves data blocks from main memory. This irregular memory access pattern fundamentally conflicts with the continuous transfer mechanism upon which DMA in the Shenwei many-core processor relies. When DMA handles non-contiguous, small-granular data transfers, the bus handshake overhead increases significantly, resulting in a sharp decline in effective bandwidth utilization and severely restricting program performance. Second, the data required by the slave core is generally stored in main memory as multidimensional arrays, and multidimensional arrays are typically stored using linear mapping in physical memory, resulting in a large span between logically adjacent data blocks in terms of physical addresses. If the data is computed directly in the slave core according to the original index order, the data blocks required by the data object will be severely scattered in memory, resulting in extremely low data reuse rate in the LDM space. This leads to frequent swapping in and out and redundant loading, causing the computation pipeline to frequently pause due to waiting for data. Third, because discrete data objects are randomly distributed, the indices of the data blocks to be retrieved from main memory are also randomly distributed. To improve computational efficiency, the entire data block needs to be stored in the LDM at once via DMA operations. However, the LDM space is limited, only 256KB in size. Even with full-array LDM contiguous shared mode enabled and a 128KB contiguous shared space capacity selected, the total space capacity reaches 8192KB (64*128KB), which is insufficient to store large-scale multidimensional array data. At this point, only direct discrete memory access (i.e., gld / gst) can be used, and the cache space capacity needs to be adjusted to speed up memory access. However, because random memory access destroys spatial locality, even with the cache space adjusted to its maximum (128KB), the final program efficiency is still not ideal. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides an optimization method for discrete data streaming memory access based on the Shenwei many-core processor; When optimizing large-scale discrete data processing programs on the Sunway many-core processor, the slave core needs to retrieve dependent background data blocks from main memory based on the characteristic attributes of discrete data objects. To mask high latency, it is common practice to attempt to transfer the computationally required data blocks to the slave core's LDM via DMA. However, global background data is typically stored in the form of large-scale multidimensional arrays, far exceeding the maximum capacity of the slave core's LDM in continuous shared mode, making it impossible to transfer the complete data to the LDM. In this case, only a general memory access method can be used, i.e., direct discrete memory access in the original order, relying on limited cache space. Due to the highly random distribution of discrete objects in physical space, this causes severe non-contiguous addressing and bus handshaking overhead, failing to fully utilize the continuous transfer characteristics of DMA, resulting in a significant waste of time. For programs exhibiting the above problems, the method of this invention can be used for optimization, effectively solving the problem and improving program execution efficiency. This invention also designs an automated interface, SIDA (Sunway Irregular Data Access), for convenient direct use by programmers.

[0009] In this invention, firstly, a spatial grid mapping benchmark is constructed based on node vector information, establishing a linear mapping relationship between discrete data objects and grid indices. Secondly, data objects are binned and globally rearranged according to the mapping relationship, ensuring that data objects with similar spatial locations are also stored in similar positions in main memory. Based on this, data objects allocated to the same slave core and processed in adjacent orders will logically exhibit a high degree of overlap or proximity in their required interpolation coefficient blocks. This enhanced data locality makes the index of the interpolation coefficient blocks more compact, reducing redundant data transfer from main memory to the slave core. This allows the slave core to efficiently load interpolation coefficient blocks into the LDM space for reuse via DMA.

[0010] This invention specifically addresses the problem of limited local storage space in the slave kernel and the random distribution of large-scale discrete data, which leads to frequent discrete accesses to the master kernel's storage resources by the slave kernel. It successfully reduces the pressure on master memory bandwidth and efficiently utilizes spatial locality, applying it to slave kernel acceleration schemes. This innovative method solves the dual bottlenecks of LDM capacity and master memory bandwidth, successfully transforming inefficient memory access to large-scale discrete data into efficient continuous burst transmissions, thus improving program execution efficiency.

[0011] The technical solution of this invention is as follows: A discrete data streaming memory access optimization method based on Shenwei many-core processor includes: Step 1: Map and rearrange discrete data in the main kernel; including: A spatial grid mapping benchmark is constructed based on node vector information, and a linear mapping relationship is established from discrete data objects to grid indices; Based on the linear mapping relationship, data objects are binned and globally rearranged so that data objects that are spatially close are stored in close proximity in main memory. Step 2: Chunk computation and split-aggregate transmission; including: By implementing data slicing and split aggregation DMA strategies from the core side, and combining this with data locality established on the main core, inefficient random memory accesses are reconstructed into efficient streaming data multiplexing.

[0012] According to a preferred embodiment of the present invention, a spatial grid mapping benchmark is constructed based on node vector information, and a linear mapping relationship is established from discrete data objects to grid indices; including: First, obtain the dimensionality information and node partition vectors of the global background reference dataset upon which the discrete data object to be processed depends. The global background reference dataset refers to the multi-dimensional mesh field data stored in main memory that covers the entire simulation space in the physical simulation. Dimensionality information refers to the number of mesh partitions in each physical dimension of the dataset. The global background reference dataset is set to be divided into... There are 1 logical grid cell, of which... The number of grid cells in the first dimension. The number of grid cells represents the second dimension; these logical grid cells define the smallest data block granularity of the computational element when performing dependent data retrieval. Then, iterate through all the computational elements to be processed. Read all computational elements to be processed Included spatial coordinate feature attributes ;in, Represents computational elements Physical coordinates in the first dimension Represents computational elements Physical coordinates in the second dimension; Based on the node partitioning vector, the node partitioning vector is an ordered array that describes the logical grid boundary positions of the global background reference dataset in each physical dimension; Determine the grid logical index to which each computation element currently belongs. ; , Each refers to a computational element The column index and row index of the logical grid cell where the current spatial location is located in the global grid system; The linear mapping relationship from discrete data objects to grid logical indices satisfies: (1); (2); in, , These refer to the grid node coordinate vectors of the global background reference dataset in the first and second dimensions, respectively. It is a collection of A one-dimensional array of elements Store the first-dimensional coordinates of the k-th grid node. It is a collection of A one-dimensional array of elements Store the second-dimensional coordinates of the k-th grid node; Further preferred, , The specific calculation formula is as follows: (3); (4); in, The function is a floor function. and These are the grid step sizes in the first and second dimensions, respectively, and the specific calculation formulas are as follows: (5); (6).

[0013] According to a preferred embodiment of the present invention, data objects are binned and globally rearranged based on a linear mapping relationship, so that data objects with similar spatial locations are stored in similar locations in main memory; including: Using either row-major or scan-line algorithms, the two-dimensional grid logical index is mapped to a one-dimensional sorted key value. The calculation formula is as follows: (7); in, This represents the number of grid cells in the first dimension. Based on the generated one-dimensional sorting key value The state data array, which includes all computational elements, is sorted globally in ascending order. The state data array is a contiguous block of memory allocated in main memory, used to store the complete physical attributes of all discrete computational elements at the current simulation moment.

[0014] According to a preferred embodiment of the present invention, the segmented computation and split-aggregate transmission includes: First, an on-demand slicing strategy is adopted, loading only the minimum state set required for the current computation step from the core, that is, using DMA to pull multiple rearranged consecutive computational elements from main memory to LDM at once; Secondly, a contiguous space is allocated in the LDM to house frequently accessed local dependency data blocks; at the same time, a variable is set as the logical index of the data block stored in the LDM, and the stored data block is the local dependency data block used to house frequently accessed data blocks. When processing the i-th computational element, the logical index of the dependent data block required by the i-th computational element is calculated and compared with the logical index of the storage data block stored in the LDM. If they are equal, the data block residing in the LDM is directly reused. If they are not equal, the starting physical address of each row of the dependent data block in main memory is calculated based on the logical index. Multiple DMA instructions are used to request independent data segments of each row and set the target address of the DMA. The data segments from different physical locations in main memory are then tightly aggregated and written into the contiguous space allocated by the LDM to eliminate memory gaps. After the transfer is complete, update the logical index stored in LDM.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described discrete data stream memory access optimization method based on the Shenwei many-core processor.

[0016] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described discrete data streaming memory access optimization method based on the Shenwei many-core processor.

[0017] An automated interface is provided for implementing the aforementioned discrete data streaming memory access optimization method based on the Shenwei many-core processor.

[0018] The beneficial effects of this invention are as follows: 1. This invention significantly improves the memory access locality of discrete data in a many-core environment and solves the bandwidth bottleneck problem caused by large-scale random memory access.

[0019] 2. This invention breaks through the hardware bottleneck of limited local memory (LDM) capacity in domestic many-core processors, greatly improving the parallelism of computing.

[0020] 3. This invention achieves a compact layout for storing non-continuous high-dimensional data on-chip, eliminating memory fragmentation and reducing redundant memory accesses. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the architecture of the Shenwei SW39000 many-core processor; Figure 2 This is a flowchart illustrating an optimization method for discrete data streaming memory access based on the Shenwei many-core processor according to the present invention. Figure 3 A diagram comparing spatial access paths before and after rearranging discrete data objects; Figure 4 This is a schematic diagram illustrating the principle of kernel-based block computation and split aggregation transmission. Detailed Implementation

[0022] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0023] Terminology Explanation: gld / gst memory access: Discrete memory access, which allows the slave kernel to directly retrieve data from or store data in main memory using the address of main memory, without using LDM as a data cache.

[0024] Cache: A smaller and faster storage device that acts as a buffer for larger and slower storage devices, thereby improving data access speed. The core can use the cache to accelerate discrete access speeds for reading or storing data from main memory.

[0025] DMA: Direct Memory Access, initiated by a single slave core, enables data transfer between LDM and main memory, improving the efficiency of slave cores accessing the main core.

[0026] LDM: Local Data Memory. Each slave core of the Shenwei many-core processor has a high-speed local data storage space (LDM). Its space capacity is limited and is a scarce resource of the processor. The LDM space is mainly divided into private space and contiguous shared space, with a total capacity of 256KB.

[0027] Example 1 A discrete data streaming memory access optimization method based on Shenwei many-core processor, such as... Figure 2 As shown, it includes: Step 1: Map and rearrange discrete data in the main kernel; including: A spatial grid mapping benchmark is constructed based on node vector information, and a linear mapping relationship is established from discrete data objects to grid indices; Based on the linear mapping relationship, data objects are binned and globally rearranged so that data objects that are spatially close are stored in close proximity in main memory. Step 2: Chunk computation and split-aggregate transmission; including: By implementing data slicing and split aggregation DMA strategies from the core side, and combining this with data locality established on the main core, inefficient random memory accesses are reconstructed into efficient streaming data multiplexing.

[0028] Example 2 The difference between the discrete data streaming memory access optimization method based on the Shenwei many-core processor described in Example 1 and the one described in Example 1 is as follows: A spatial grid mapping baseline is constructed based on node vector information, establishing a linear mapping relationship from discrete data objects to grid indices; including: Before performing calculations on the data from the kernel, it is first mapped and rearranged in the main kernel. First, the dimensionality information and node partitioning vectors of the global background reference dataset upon which the discrete data object to be processed depends are obtained. The global background reference dataset refers to the multi-dimensional mesh field data stored in main memory that covers the entire simulation space in the physical simulation; for example, two-dimensional or three-dimensional electromagnetic field mesh data in plasma simulations. The discrete data object needs to read local data from this dataset for interpolation or updating during calculation. Dimensional information refers to the number of mesh partitions in each physical dimension of the dataset; in this example, the global background reference dataset is set to be divided into... A logical grid, in which The number of grid cells in the first dimension (such as the R-axis or X-axis). This refers to the number of grid cells in the second dimension (such as the Z-axis or Y-axis). These logical grid cells define the smallest data block granularity for computational elements when performing dependent data retrieval.

[0029] Then, iterate through all the computational elements to be processed. Read all computational elements to be processed Included spatial coordinate feature attributes ;in, Represents computational elements Physical coordinates in the first dimension (e.g., R coordinates). Represents computational elements Physical coordinates in the second dimension (such as the Z coordinate); and The reading process is as follows: The program pre-allocates a structure array in the global main memory to store the state information of all computation elements, which contains the coordinate information of the computation elements. When calling the interface in the main core, the pointer of this structure array is passed as a parameter.

[0030] Based on the node partitioning vector, which is an ordered array describing the logical grid boundary positions of the global background reference dataset in each physical dimension; in this example, it contains two vectors: the first dimension node vector ( ), corresponding to the R-axis, including Each element records the boundary within the simulation region. to the outer boundary The physical coordinates of all grid nodes; the second-dimensional node vector ( ), corresponding to the Z-axis, including Each element records the distance from the lower boundary of the simulation region. to the upper boundary The physical coordinates of all grid nodes. These two vectors together constitute a grid system, dividing the continuous physical space into small rectangular logical units.

[0031] Acquisition process: Calculated using linear interpolation formula: ; ; Determine the grid logical index to which each computation element currently belongs. ; , Each refers to a computational element The column and row indices of the logical grid cell where the current spatial location is located within the global grid system; these two indices are non-negative integers, with values ​​ranging from [value range missing]. and They uniquely identify the logical location of the dependent data blocks required by the computational element within the global background dataset.

[0032] The linear mapping relationship from discrete data objects to grid logical indices satisfies: (1); (2); in, , These refer to the grid node coordinate vectors of the global background reference dataset in the first and second dimensions, respectively. It is a collection of A one-dimensional array of elements Store the first-dimensional coordinates of the k-th grid node. It is a collection of A one-dimensional array of elements Stores the second-dimensional coordinate value of the k-th grid node; this index is directly associated with the logical base address in global main memory of the local dependent data block required by the computation element.

[0033] , The specific calculation formula is as follows: (3); (4); in, The function is a floor function. and These are the grid step sizes in the first and second dimensions, respectively, and the specific calculation formulas are as follows: (5); (6).

[0034] Binning and global rearrangement of data objects based on linear mapping relationships ensures that data objects with similar spatial locations are stored close together in main memory; this includes: To adapt to the linear one-dimensional storage structure of computer memory, row-major order or a similar scan-line algorithm is used to map the two-dimensional grid logical index to a one-dimensional sorted key value. The calculation formula is as follows: (7); in, This represents the number of grid cells in the first dimension; it is a constant set by the user after testing.

[0035] This formula ensures that adjacent computational units in a multidimensional space generate key values ​​that exhibit local continuity or monotonically increasing numerical characteristics. Based on the generated one-dimensional sorted key values... The state data array, which includes all computational elements, is globally sorted in ascending order. The state data array is a contiguous block of memory allocated in main memory, used to store the complete physical attributes of all discrete computational elements at the current simulation moment. Each element in the array corresponds to a computational element, and this element contains a composite data structure that encapsulates information such as the computational element's index, spatial coordinates, velocity components, and physical weights. The state data array is quickly sorted using the qsort function provided in the C standard library, which serves as the sorting key.

[0036] like Figure 3 As shown, after the rearrangement, computational elements whose spatial locations fall into the same or adjacent grids are reorganized into contiguous storage at physical memory addresses. This lays the foundation for data locality for subsequent streaming processing from the kernel.

[0037] Block-based computation and split-aggregate transmission; such as Figure 4 As shown, it includes: By rearranging discrete data objects, their physical proximity is successfully mapped to memory address contiguity, eliminating the randomness of data distribution. However, two problems still exist for the Shenwei many-core processor architecture. First, the contradiction between massive ordered data and the tiny LDM capacity of the slave core prevents all of it from residing there. Second, the conflict between the non-contiguous nature of multidimensional dependent data in physical memory and the linear DMA transfer mechanism. Therefore, this invention proposes an innovative solution: solving the LDM capacity limitation problem on the slave core side through data slicing, solving the physical address discontinuity problem using a split-aggregate DMA strategy, and combining this with establishing data locality on the master core to reconstruct inefficient random memory access into efficient streaming data multiplexing.

[0038] First, an on-demand slicing strategy is adopted. The slave core loads only the minimum state set required by the computational element in the current calculation step (i.e., only containing the spatial coordinates and necessary feature attributes at the current moment), filtering out unnecessary historical data. Based on this slicing strategy, the minimum state set refers to the minimum set of data fields required for the current slave core to complete the calculation during program execution. According to the needs of the current slave core function, only specific member variables are extracted from the complete data structure in main memory and transferred to the slave core LDM, filtering out redundant data that does not need to be accessed in the current step. For example, when the slave core performs electromagnetic field interpolation calculations, it only depends on the spatial location of the computational element, so the slave core only needs to retrieve the spatial location of the computational element from main memory, without needing other attributes. The slave core uses DMA to pull multiple rearranged consecutive computational elements from main memory to the LDM at once. The consecutive computational elements are determined after rearrangement in step 1. Based on the total number of slave cores (e.g., 64), the total array of rearranged computational elements is evenly divided into 64 consecutive index intervals, and each slave core is only responsible for processing one interval corresponding to its ID.

[0039] Secondly, a contiguous space is allocated in the LDM to house frequently accessed locally dependent data blocks. Simultaneously, a variable is established as the logical index for storing data blocks in the LDM; these stored data blocks are the frequently accessed locally dependent data blocks. The index value stored in this variable strictly corresponds to the local dependent data block actually residing in the LDM at the current moment. For example, assuming 1000 grids of magnetic field data are stored in main memory, a space of size that can only store one grid of magnetic field data is allocated in the LDM. A variable is defined in the LDM to store the grid index of the magnetic field data currently stored in the LDM and initialized to -1.

[0040] When processing the i-th computational element, the logical index of the dependent data block required by the i-th computational element is calculated. In this example, the program pre-sets two one-dimensional monotonically increasing arrays in main memory: a radial grid node coordinate array and a vertical grid node coordinate array. The program performs a binary search in these two arrays using the R and Z coordinates of the current computational element to determine the geometric interval index where the coordinate falls, which is the logical index of the dependent data block required by the current computational element. The calculated logical index is then compared with the logical index of the stored data block in the LDM. If they are equal, the data block residing in the LDM is directly reused. If they are not equal, the starting physical address of each row of the dependent data block in main memory is calculated based on the logical index. Multiple DMA instructions are used to request independent data segments for each row, and the target address of the DMA is set. Data segments from different physical locations in main memory are then tightly aggregated and written into the contiguous space allocated by the LDM, eliminating memory gaps.

[0041] Based on the logical index of the required dependent data block, the starting physical address of each row of data fragments in the main memory is calculated. Addressing the physical discontinuity of the dependent data block in main memory due to row-major storage, the slave core does not initiate a single large block transfer. Instead, it constructs multiple independent DMA transfer instructions, each dedicated to moving a single row of valid data from the matrix, skipping irrelevant data between rows. Furthermore, when configuring the DMA target address, a continuous linear address in the LDM is specified. Through this "discrete read, continuous write" address remapping mechanism, multiple data rows that were originally separated by huge spans in main memory are tightly concatenated and written into the continuous space of the LDM, physically eliminating the original memory gaps. This achieves a transformation from sparse to compact data layout, greatly improving the space utilization of the LDM and the memory access continuity of subsequent calculations.

[0042] After the transmission is complete, update the logical index stored in the LDM to enable subsequent computational reuse.

[0043] In this way, within the same processing batch, thanks to the high locality brought about by the rearrangement, the frequency of the "split-aggregate transmission" operation is greatly reduced, achieving the streaming processing effect of "load once, reuse multiple times" and breaking through the dual bottlenecks of main memory bandwidth and on-chip capacity.

[0044] The method described above was used to test the method of the present invention within a single core group.

[0045] This test uses the plasma guidance center orbit simulation program as an example. Under the same conditions, three data volumes were set according to the number of particles: small scale, medium scale, and large scale, which were 1280, 12800, and 38400, respectively. The acceleration ratio was tested at each of these three scales.

[0046] To accurately evaluate the speedup effect of this invention, three program versions with different optimization levels were selected for comparative testing: a serial program version, a general many-core optimized program version, and an invention-optimized program version. The serial program refers to a program that executes only on a single thread on the main core; the general many-core optimized program refers to a program that, although utilizing parallel computation on slave cores, uses inefficient direct discrete memory access instructions (i.e., gld / gst) to obtain data; and the invention-optimized program is a program that fully applies the method described in this invention and the SIDA interface. The time and speedup ratio of the three programs were compared. Experimental results demonstrate that the method described in this invention has a significant speedup effect compared to the serial program and the general many-core optimized program. Specific results and comparisons are shown in the table below.

[0047] Table 1 shows the runtime and speedup ratio of the test program before and after optimization when the particle number is 1280; Table 1 is as follows: Table 1. Acceleration ratio when the particle number is 1280; Table 2 shows the runtime and speedup ratio of the test program before and after optimization when the particle number is 12800; Table 2 is as follows: Table 2 Acceleration ratio when the particle number is 12800; Table 3 shows the runtime and speedup ratio of the test program before and after optimization when the particle number is 38400; Table 3 is as follows: Table 3 Acceleration ratio when the particle number is 38400; Example 3 A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the discrete data stream memory access optimization method based on the Shenwei many-core processor described in Embodiment 1 or 2.

[0048] Example 4 A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the discrete data streaming memory access optimization method based on Shenwei many-core processor described in Embodiment 1 or 2.

[0049] Example 5 An automated interface for implementing the discrete data streaming memory access optimization method based on Shenwei many-core processor described in Embodiment 1 or 2.

[0050] This invention also designs and implements the SIDA (Self-Automatic Interface), which aims to solve the complex memory access management challenges of large-scale discrete data under the Sunway many-core architecture. The interface automates the aforementioned spatial grid-based discrete data rearrangement, on-demand slicing and batch loading of slave data, and split-aggregate DMA transfer strategies. The implementation of this interface reduces the difficulty of programming on the Sunway supercomputer, allowing programmers to directly call this interface to solve problems. The interface consists of two parts: the master core and the slave core. Interface descriptions are shown in Table 4.

[0051] Table 4 SIDA Interface Description; When calling the SIDA interface, programmers can set the interface parameters according to the specific program.

[0052] The following is an example of the main core calling the SIDA interface: / / Call the SIDA interface master.c in the main core. #include “SIDA.h” int main() { / / Define the variables required for the physics simulation / / Initialize the SIDA interface environment SIDA_Init(); athread_init(); / / Serial code segment / / Perform data rearrangement preprocessing; SIDA_Reorder_Map(init_p,init_p_size,r_max,r_min,z_max,z_min); / / Start parallel computation from the core array athread_spawn(slave_kernel,&init_p); / / Wait for calculations to finish and clean up. athread_join(); SIDA_Finalize(); return 0; } An example of calling the SIDA interface from the kernel is as follows: / / Calling the SIDA interface slave.c from the kernel #include “SIDA.h” #include<slave.h> void slave_kernel(Particle *l_p_m) { / / Define the required variables / / Obtain the structure pointer from the main kernel via DMA Particle l_p_s; athread_dma_get(&l_p_s, l_p_m, sizeof(Particle)); / / Get the current slave core ID int id = _PEN; / / Get the index range of particles that need to be processed from the kernel. / / Based on the rearrangement results in the master nucleus, assign a continuous segment of particles to each slave nucleus. int p_start, p_end; Get_Thread_Range(id, p_start, p_end); / / Batch acquisition of particle RZ coordinates via DMA athread_dma_get(p_loc_r, l_p_s.p_loc_r+p_start, (p_end-p_start+1)*sizeof(double)); athread_dma_get(p_loc_z, l_p_s.p_loc_z+p_start, (p_end-p_start+1)*sizeof(double)); / / Main calculation loop for(int i = 0; i<(p_end-p_start+1); i++) { / / Get dependent data blocks SIDA_Get_Stream_Data(p_loc_r[i],p_loc_z[i],rely_data,rely_ptr,index_n); / / Perform calculations using the obtained dependency data block pointer, rely_ptr. / / Hot spots in the program / / Calculation complete } }

Claims

1. A method for optimizing discrete data stream memory access based on a Sunway many-core processor, characterized in that, include: Step 1: Map and rearrange discrete data in the main kernel; including: A spatial grid mapping baseline is constructed based on node vector information, establishing a linear mapping relationship from discrete data objects to grid indices; including: First, obtain the dimensionality information and node partition vectors of the global background reference dataset upon which the discrete data object to be processed depends. The global background reference dataset refers to the multi-dimensional mesh field data stored in main memory that covers the entire simulation space in the physical simulation. Dimensionality information refers to the number of mesh partitions in each physical dimension of the dataset. The global background reference dataset is set to be divided into... There are 10 logical grid cells, of which... The number of grid cells in the first dimension. The number of grid cells represents the second dimension; these logical grid cells define the smallest data block granularity of the computational element when performing dependent data retrieval. Then, iterate through all the computational elements to be processed. Read all computational elements to be processed Included spatial coordinate feature attributes ;in, Represents computational elements Physical coordinates in the first dimension Represents computational elements Physical coordinates in the second dimension; Based on the node partitioning vector, the node partitioning vector is an ordered array that describes the logical grid boundary positions of the global background reference dataset in each physical dimension; Determine the grid logical index to which each computation element currently belongs. ; , Each refers to a computational element The column index and row index of the logical grid cell where the current spatial location is located in the global grid system; The linear mapping relationship from discrete data objects to grid logical indices satisfies: （1）； （2）； in, , These refer to the grid node coordinate vectors of the global background reference dataset in the first and second dimensions, respectively. It is a collection of A one-dimensional array of elements Store the first-dimensional coordinates of the k-th grid node. It is a collection of A one-dimensional array of elements Store the second-dimensional coordinates of the k-th grid node; Based on the linear mapping relationship, data objects are binned and globally rearranged so that data objects that are spatially close are stored in close proximity in main memory. Step 2: Chunk computation and split-aggregate transmission; including: By implementing data slicing and split-aggregate DMA strategies from the core side, and combining this with data locality established on the main core, inefficient random memory access is reconstructed into efficient streaming data multiplexing; including: First, an on-demand slicing strategy is adopted, loading only the minimum state set required for the current computation step from the core, that is, using DMA to pull multiple rearranged consecutive computational elements from main memory to LDM at once; Secondly, a contiguous space is allocated in the LDM to house frequently accessed local dependency data blocks; at the same time, a variable is set as the logical index of the data block stored in the LDM, and the stored data block is the local dependency data block used to house frequently accessed data blocks. When processing the i-th computational element, the logical index of the dependent data block required by the i-th computational element is calculated and compared with the logical index of the storage data block stored in the LDM. If they are equal, the data block residing in the LDM is directly reused. If they are not equal, the starting physical address of each row of the dependent data block in main memory is calculated based on the logical index. Multiple DMA instructions are used to request independent data segments of each row and set the target address of the DMA. The data segments from different physical locations in main memory are then tightly aggregated and written into the contiguous space allocated by the LDM to eliminate memory gaps. After the transfer is complete, update the logical index stored in LDM.

2. The discrete data streaming memory access optimization method based on Shenwei many-core processor according to claim 1, characterized in that, , The specific calculation formula is as follows: （3）； （4）； in, The function is a floor function. and These are the grid step sizes in the first and second dimensions, respectively, and the specific calculation formulas are as follows: （5）； （6）。 3. The discrete data streaming memory access optimization method based on Shenwei many-core processor according to claim 1, characterized in that, Binning and global rearrangement of data objects based on linear mapping relationships ensures that data objects with similar spatial locations are stored close together in main memory; this includes: Using either row-major or scan-line algorithms, the two-dimensional grid logical index is mapped to a one-dimensional sorted key value. The calculation formula is as follows: （7）； in, This represents the number of grid cells in the first dimension. Based on the generated one-dimensional sorting key value The state data array, which includes all computational elements, is sorted globally in ascending order. The state data array is a contiguous block of memory allocated in main memory, used to store the complete physical attributes of all discrete computational elements at the current simulation moment.

4. An automation interface, characterized in that, This method is used to implement the discrete data streaming memory access optimization method based on Shenwei many-core processor as described in any one of claims 1-3.

Images