Memory access method and electronic device

Abstract

The application provides a memory access method and an electronic device, and applies to the technical field of accelerator system. The memory access method comprises the following steps: determining a target memory access strategy from a preset memory access strategy set according to the local memory access characteristics of any computing core; in the case that the target memory access strategy is a transmission optimization strategy, determining the position information of the to-be-accessed data in the target cache line according to the target cache line address in the initial memory access request initiated by any computing core; generating a bit mask for identifying the to-be-accessed data according to the position information; adding the bit mask to the initial memory access request to form a memory access request, and sending the memory access request to a target device to which the target cache line address belongs; and the target device returns the to-be-accessed data according to the bit mask in the memory access request.

Description

Technical Field

[0001] This application relates to the field of multi-accelerator system technology, and more specifically to a memory access method and electronic device. Background Technology

[0002] In a multi-accelerator system, each accelerator (such as a graphics processing unit or an AI accelerator) has its own local memory (e.g., high-bandwidth memory (HBM)). These accelerators access each other's remote memory via interconnect links, enabling cross-device memory access. In this architecture, local memory access can fully utilize the high bandwidth of HBM (typically reaching hundreds of GB / s), while remote memory access is limited by the bandwidth and latency of the interconnect links. This difference in access performance leads to severe performance bottlenecks in multi-accelerator systems for computational workloads that heavily rely on remote data (such as breadth-first search and sparse matrix-vector multiplication). Summary of the Invention

[0003] In view of the above problems, this application provides a memory access method and an electronic device.

[0004] According to a first aspect of this application, a memory access method is provided, applied to a multi-accelerator system and executed by any computing core in the multi-accelerator system. The method includes: determining a target memory access strategy from a preset memory access strategy set based on the local memory access characteristics of the computing core; if the target memory access strategy is a transmission optimization strategy, determining the location information of the data to be accessed in the target cache line based on the target cache line address in the initial memory access request initiated by the computing core; generating a bitmask for identifying the data to be accessed based on the location information, wherein the bitmask represents the data block information occupied by the data to be accessed among multiple data blocks obtained by dividing the target cache line according to a preset data block size; adding the bitmask to the initial memory access request to form a memory access request, and sending the memory access request to the target device to which the target cache line address belongs, wherein the target device returns the data to be accessed based on the bitmask in the memory access request.

[0005] A second aspect of this application provides a memory access device applied to a multi-accelerator system, executed by any computing core in the multi-accelerator system. The device includes: a first determining module, configured to determine a target memory access strategy from a preset memory access strategy set based on the local memory access characteristics of any computing core; a second determining module, configured to determine the location information of the data to be accessed in the target cache line based on the target cache line address in the initial memory access request initiated by any computing core, when the target memory access strategy is a transmission optimization strategy; a generating module, configured to generate a bitmask for identifying the data to be accessed based on the location information, wherein the bitmask represents the data block information occupied by the data to be accessed among multiple data blocks obtained by dividing the target cache line according to a preset data block size; and a first sending module, configured to add the bitmask to the initial memory access request to form a memory access request, and send the memory access request to the target device to which the target cache line address belongs, wherein the target device returns the data to be accessed based on the bitmask in the memory access request.

[0006] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0007] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0008] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0009] This application enables each computing core to autonomously perceive its memory access characteristics. By sensing local memory access characteristics in real time, it adaptively selects an appropriate memory access strategy based on this. Specifically, when the memory access strategy is a fine-grained transmission optimization strategy, bitmasks are used to accurately identify the required data blocks, thereby avoiding invalid data transmission caused by traditional cache line-level transmission methods. This reduces bandwidth waste and transmission latency in interconnect links, and ultimately helps improve the overall data throughput efficiency and computing resource utilization of multi-accelerator systems. Attached Figure Description

[0010] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0011] Figure 1The illustration shows application scenarios of memory access methods, apparatuses, electronic devices, storage media, and program products according to embodiments of this application.

[0012] Figure 2 A flowchart of a memory access method according to an embodiment of this application is shown.

[0013] Figure 3 A flowchart illustrating a method for determining a target memory access strategy based on an embodiment of this application is shown.

[0014] Figure 4 A schematic diagram of the architecture of an accelerator according to an embodiment of this application is shown.

[0015] Figure 5 A schematic diagram of a bitmask according to an embodiment of this application is shown.

[0016] Figure 6 A schematic diagram illustrating the implementation logic of a cache bypass unit according to an embodiment of this application is shown.

[0017] Figure 7 A schematic diagram illustrating request matching using a cached request missing status holding register according to an embodiment of this application is shown.

[0018] Figure 8 A schematic diagram is shown where the address and bitmask match according to an embodiment of this application.

[0019] Figure 9 A schematic diagram is shown where the address matches but the bitmask does not match according to an embodiment of this application.

[0020] Figure 10 A schematic diagram is shown where the address and bitmask do not match according to an embodiment of this application.

[0021] Figure 11 A schematic diagram of a remote direct memory access module according to an embodiment of this application is shown.

[0022] Figure 12 A schematic diagram of the process of remote direct memory access recoded storage according to an embodiment of this application is shown.

[0023] Figure 13 A structural block diagram of a memory access device according to an embodiment of this application is shown.

[0024] Figure 14 A block diagram of an electronic device suitable for implementing a memory access method according to an embodiment of this application is shown. Detailed Implementation

[0025] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] With Moore's Law nearing its end, single-accelerator systems are struggling to meet the dual demands of emerging workloads (such as large-scale AI training and sparse matrix operations) for both computing power and memory capacity. To address this, the industry has proposed tightly coupled multi-accelerator systems, which leverage the collaborative work of multiple accelerators to expand computing and storage resources.

[0030] However, the performance potential of such systems has not been fully realized in practical applications. The core bottleneck lies in the inefficient remote memory access caused by the Non-Uniform Memory Access Architecture (NUMA).

[0031] A multi-accelerator system refers to a computing system consisting of multiple accelerators (such as GPUs and AI accelerators) with independent computing and memory resources connected by a high-speed interconnect network. In a multi-accelerator system, each accelerator has its own local memory (such as High Bandwidth Memory, HBM) and can access the remote memory of other accelerators through interconnect links such as Peripheral Component Interconnect Express (PCIe), NVIDIA NVLink, and Universal Chiplet Interconnect Express (UCIE). Under this architecture, local memory access can fully utilize the high bandwidth of HBM, while remote memory access is limited by the bandwidth and latency of the interconnect links, resulting in performance limitations for workloads that rely on remote data.

[0032] For example, in a system with two accelerators, when accelerator 0 needs to access the memory of accelerator 1, the data must go through a complete process of "local accelerator request initiation → interconnect link transmission → remote accelerator data reading → link return," with the link transmission stage alone accounting for 80% of the total latency of remote memory access. This latency difference limits the performance of multi-accelerator systems when running workloads that rely on remote data (such as breadth-first search and sparse matrix-vector multiplication).

[0033] To alleviate the NUMA bottleneck, existing research and practice mainly adopt two methods: page migration and data copying.

[0034] Page migration schemes dynamically adjust the ownership of memory pages at runtime, migrating frequently accessed remote pages to local accelerators. While this method can balance page distribution to some extent, the migration process requires clearing the caches, translation lookaside buffers (TLBs), and computation unit pipelines of remote accelerators, resulting in significant performance overhead. Furthermore, for pages shared by multiple accelerators, a "ping-pong effect" occurs (i.e., pages are repeatedly migrated between multiple accelerators), reducing system efficiency.

[0035] Data replication solutions copy remote data to local memory to avoid repeated remote access to the same data. However, this approach requires additional local storage resources, and for workloads with large working sets (such as large-scale matrix operations), the storage overhead may outweigh the performance gains. Furthermore, some solutions require programmers to manually mark the data to be copied, increasing programming complexity and making it difficult to flexibly adapt to diverse workloads.

[0036] Based on this, embodiments of this application provide a memory access method applied to a multi-accelerator system, executed by any computing core in the multi-accelerator system. The method includes: determining a target memory access strategy from a preset memory access strategy set based on the local memory access characteristics of any computing core; if the target memory access strategy is a transmission optimization strategy, determining the location information of the data to be accessed in the target cache line based on the target cache line address in the initial memory access request initiated by any computing core; generating a bitmask to identify the data to be accessed based on the location information, wherein the bitmask represents the data block information occupied by the data to be accessed among multiple data blocks obtained by dividing the target cache line according to a preset data block size; adding the bitmask to the initial memory access request to form a memory access request, and sending the memory access request to the target device to which the target cache line address belongs; the target device returning the data to be accessed based on the bitmask in the memory access request. This method achieves real-time perception of local memory access characteristics and adaptive selection of an appropriate memory access strategy based on these characteristics. Furthermore, when the memory access strategy is a fine-grained transmission optimization strategy, bitmasks are used to accurately identify the required data blocks, thereby avoiding invalid data transmission caused by the traditional cache line-level transmission method. This reduces bandwidth waste and transmission latency in interconnect links, which in turn helps to improve the overall data throughput efficiency and computing resource utilization of multi-accelerator systems.

[0037] Figure 1 The illustration shows application scenarios of memory access methods, apparatuses, electronic devices, storage media, and program products according to embodiments of this application.

[0038] like Figure 1 As shown, the memory access method of this embodiment can be applied to a multi-accelerator system 100, which may include a first accelerator 110, a second accelerator 120, and a third accelerator 130, all of which are connected by interconnect links. Each accelerator includes multiple computing cores (e.g., a first computing core, a second computing core, ..., an Nth computing core, where N≥3), and each computing core can independently execute the following memory access method.

[0039] Taking the first computing core in the first accelerator 110 as an example, the first computing core can monitor its own local access characteristics 101 in real time and determine the target memory access strategy from the preset memory access strategy set 102 based on the local memory access characteristics 101. If the target memory access strategy is a transmission optimization strategy, the first computing core 111 can determine the location information 105 of the data to be accessed in the target cache line based on the target cache line address 104 in its own initial memory access request 103 (for example, the target cache line address belongs to the second accelerator); then, based on the location information 105, it generates a bitmask 106 to identify the data to be accessed, wherein the bitmask 106 represents the data block information occupied by the data to be accessed in the multiple data blocks obtained by dividing the target cache line according to the preset data block size; then, the bitmask 106 is added to the initial memory access request 103 to form a memory access request 107, and the memory access request 107 is sent to the second accelerator 120.

[0040] After receiving the memory access request 107, the second accelerator 120 extracts the data to be accessed according to the bitmask 106 in the memory access request 107 and returns it to the first accelerator 110.

[0041] It should be understood that Figure 1 The number of the first, second, and third accelerators shown is merely illustrative. Any number of accelerators can be used depending on implementation requirements.

[0042] The following will be based on Figure 1 The described scene, through Figures 2-12 The memory access method of the disclosed embodiments will be described in detail.

[0043] Figure 2 A flowchart of a memory access method according to an embodiment of this application is shown.

[0044] like Figure 2 As shown, the memory access method of this embodiment can be applied to a multi-accelerator system and executed by any computing core in the multi-accelerator system. The memory access method of this embodiment includes operations S210 to S240.

[0045] In operation S210, a target memory access strategy is determined from a set of preset memory access strategies based on the local memory access characteristics of any computing core.

[0046] A computing core can be a basic execution unit inside an accelerator, capable of independently initiating memory access requests.

[0047] Local memory access characteristics refer to a set of quantitative indicators extracted from the perspective of a single computing core by monitoring its memory access behavior over a short period of time, which can characterize its data access patterns. For example, local memory access characteristics may include at least one of the following: remote access ratio, cache line utilization, cache hit rate, etc.

[0048] The remote access ratio refers to the proportion of memory access requests involving remote addresses among L1 cache requests. This metric is used to identify and filter "remote-dominated loads" that need optimization. It should be noted that the remote addresses here include the memory addresses of other compute cores within the same accelerator, as well as the memory addresses of other accelerators in a multi-accelerator system.

[0049] Cache line utilization is calculated by using the bitmask generation unit to count the number of data blocks actually used in remote memory access requests and to calculate the proportion of the total cache line capacity. This metric reflects the spatial locality of memory access.

[0050] The bitmask generation unit evenly divides the 64-byte cache line (the accelerator's default cache line size) into 16 4-byte data blocks (64B ÷ 4B = 16), and assigns a flag bit to each data block. These flag bits together form a 16-bit bitmask. In this bitmask, the i-th bit (i starts counting from 0) corresponds to the i-th 4-byte data block within the cache line. If a data block is accessed by a thread (i.e., it belongs to valid data), the corresponding flag bit is set to 1; otherwise, it remains 0.

[0051] For example, if the total number of data blocks in a cache line is 16, and a cache line uses 16 data blocks (determined by the number of "1"s in the bitmask), then the utilization rate of that cache line is 100%. If another cache line uses 2 data blocks, then the utilization rate of that cache line is 12.5%.

[0052] After determining the cache line utilization of each cache line, the average cache line utilization of each cache line can be taken to obtain the average cache line utilization of the computing core.

[0053] L1 cache hit rate statistics show the proportion of remote memory access requests that are hit in the L1 cache, reflecting the temporal locality of the workload. The specific statistical rule is: the number of valid cache lines corresponding to remote addresses stored in the L1 cache is divided by the total number of L1 cache lines.

[0054] To balance decision latency and classification accuracy, the monitoring window was set to "the first 25,000 remote memory access requests". Experiments show that this window length accounts for only about 0.1% of the total application execution time, and its statistical results are sufficient to accurately reflect the global locality characteristics of the workload.

[0055] A preset memory access strategy set refers to a predefined set of memory access optimization strategies, which may include, for example, transfer optimization strategies and remote data caching strategies.

[0056] Determining a target memory access strategy from a preset memory access strategy set based on the local memory access characteristics of any computing core may include: determining the workload type based on the remote access ratio, cache line utilization, L1 cache hit rate, and a preset threshold, and determining the corresponding memory access strategy based on the workload type.

[0057] Workload types include cache-insensitive workloads and cache-intensive workloads.

[0058] Cache-insensitive workloads refer to a type of computational workload that benefits little or no from caching. Their characteristic is that the probability of repeatedly accessing the same data is very low. Therefore, storing data in a cache does not effectively reduce latency for subsequent accesses; instead, it may introduce additional costs due to cache management overhead (such as filling, replacement, and consistency maintenance).

[0059] Cache-intensive workloads refer to a class of computational tasks that heavily rely on caches and can derive significant performance benefits from them. Their core characteristic is good data access locality, which allows caches to efficiently store and reuse frequently accessed data, thereby avoiding repeated accesses to slow main memory.

[0060] The threshold settings are based on statistical analysis of 11 types of load: cache line utilization for cache-insensitive loads is generally below 15% and hit rate is below 30%, while at least one of the two indicators for cache-intensive loads is above 25%. Therefore, the threshold corresponding to cache line utilization is set to 20%, and the threshold corresponding to L1 cache hit rate is set to 40%, which can effectively distinguish between the two types of loads.

[0061] The following combination Figure 3 The method for determining the target memory access strategy is explained.

[0062] Figure 3 A flowchart illustrating a method for determining a target memory access strategy based on an embodiment of this application is shown.

[0063] like Figure 3 As shown, the method for determining the target memory access strategy in this embodiment includes operations S310 to S380.

[0064] When operating the S310, the compute core monitors its local memory access characteristics in real time. These characteristics include the proportion of remote accesses, cache line utilization, and cache hit rate.

[0065] In operation S320, determine whether the remote access ratio is less than 2%. If the remote access ratio is less than 2%, then execute operation S390; if the remote access ratio is greater than or equal to 2%, then execute operation S330.

[0066] In operation S330, determine whether the cache line utilization rate is greater than 20%. If the cache line utilization rate is greater than 20%, then execute operations S340 and S350; if the cache line utilization rate is less than or equal to 20%, then execute operation S360.

[0067] When operating S340, it was determined that the workload of the computing core was cache-intensive.

[0068] When operating the S350, the target memory access strategy for the computing core is determined to be a remote data caching strategy.

[0069] In operation S360, determine whether the L1 cache hit rate is greater than 40%. If the L1 cache hit rate is greater than 40%, then execute operations S340 and S350; if the L1 cache hit rate is less than or equal to 40%, then execute operations S370 and S380.

[0070] When operating the S370, it was determined that the workload of the computing core was a cache-insensitive workload.

[0071] When operating S380, the target memory access strategy for the computing core is determined to be a transfer optimization strategy.

[0072] When operating S390, the workload of the compute core is determined to be locally dominant. No optimization strategies are enabled for locally dominant workloads.

[0073] It should be noted that the decision engine of the computing core enters the optimization phase after the monitoring phase ends, and continues to monitor the load characteristics. If the locality of the load changes abruptly (such as load switching in a multi-tenant scenario), the monitoring phase is triggered again and the optimization strategy is updated.

[0074] The transmission optimization strategy addresses the issues of "low cache utilization" and "high bandwidth waste" in cache-insensitive workloads. It employs a two-step optimization approach—"fine-grained data transmission + request merging"—to reduce invalid data transmission on interconnect links while avoiding cache overhead. Specifically, this includes operations S220 to S240.

[0075] In operation S220, when the target memory access strategy is a transmission optimization strategy, the location information of the data to be accessed in the target cache line is determined based on the target cache line address in the initial memory access request initiated by any computing core.

[0076] Location information refers to the byte offset and length of the data to be accessed within its respective cache line (i.e., the target cache line). The byte offset refers to the byte distance from the starting address of the data to the starting address of the cache line. The data length refers to the number of consecutive bytes that need to be accessed.

[0077] Location information can be obtained by extracting bits [5:2] of the target cache line address.

[0078] In operation S230, a bitmask is generated based on the location information to identify the data to be accessed. The bitmask represents the data block information occupied by the data to be accessed in the multiple data blocks obtained by dividing the target cache line according to the preset data block size.

[0079] A bitmask is a binary code in which each bit corresponds to a data block (such as a 4-byte data block) within a cache line, used to identify which data blocks in the cache line are accessed by the current request.

[0080] Specifically, a cache line (e.g., 64 bytes) is pre-divided into several fixed-size data blocks (e.g., 4 bytes per block, 16 blocks in total). Correspondingly, a bitmask consists of the same number of binary bits (16 bits in this example), each bit corresponding to one data block sequentially. When a data block is overwritten by a current memory access request (i.e., the request accesses any byte within that data block), the corresponding bit in the bitmask is set to 1; otherwise, it is 0.

[0081] In operation S240, a bitmask is added to the initial memory access request to form a memory access request, and the memory access request is sent to the target device to which the target cache line address belongs. The target device returns the data to be accessed based on the bitmask in the memory access request.

[0082] Adding a bitmask to the initial memory access request to form a memory access request can include: adding the bitmask to the target cache line address and then transmitting it using reserved bits in the initial memory access request structure. This design can transmit fine-grained data request information while avoiding the introduction of additional bandwidth overhead.

[0083] The target device refers to the physical device to which the target cache line address belongs. It can be other computing cores within the local accelerator or other accelerators.

[0084] Upon receiving a memory access request, the target device does not read and return the entire cache line (e.g., 64 bytes) as in the traditional manner. Instead, it parses the bitmask attached to the memory access request and only reads, prepares, and returns the specific data blocks marked as 1 in the bitmask. For example, if the bitmask indicates that only 3 out of 16 data blocks are needed, the target device will only return the valid data for those 3 blocks (a total of 12 bytes).

[0085] According to embodiments of this application, a target memory access strategy is dynamically selected based on local memory access characteristics (such as remote access ratio, cache line utilization, and L1 cache hit rate). If the strategy is a transmission optimization strategy, the byte offset and length of the data to be accessed are determined according to the target address of the memory request, and a bitmask identifying the specific data block is generated. This bitmask is appended to the request via reserved bits in the request frame and sent to the target device. After parsing the bitmask, the target device returns only the marked data block, not the entire cache line. This method reduces latency and bandwidth waste in remote memory access by transmitting data on demand with fine-grained granularity, avoiding additional bandwidth overhead.

[0086] The following is combined Figure 4 The memory access method under the transmission optimization strategy is further explained.

[0087] Figure 4 A schematic diagram of the architecture of an accelerator according to an embodiment of this application is shown.

[0088] like Figure 4 As shown, the accelerator in this embodiment (such as accelerator 0) includes multiple computing cores (such as a first computing core, a second computing core, and a third computing core), a cross-connect switch, and a remote RDMA codec module. RDMA stands for Remote Direct Memory Access.

[0089] Each computing core includes a cache line merging unit, an L1 cache with an integrated cache bypass unit, a cache request missing status holding register, and an L2 cache.

[0090] The L1 caches of each computing core are interconnected with each other, as are the L1 caches and L2 caches, and each computing core is interconnected with the remote RDMA codec module via cross-connect switches.

[0091] In related examples, a cache line merging unit (such as a merger) is used to merge memory access requests initiated by the computing core into 64-byte cache line addresses. However, in this embodiment, the cache line merging unit adds a bitmask generation unit, which is used to attach a 16-bit bitmask to each cache line address, where each bit corresponds to one 4-byte data block in the cache line (a total of 16 data blocks in a 64-byte cache line).

[0092] The cache line merging unit in this application is responsible for merging memory access requests initiated by the computing core and generating bitmasks.

[0093] Specifically, in operation S230, generating a bitmask to identify the data to be accessed based on the location information includes: determining at least one data block in the target cache line corresponding to the data to be accessed based on the byte offset, data length, and preset data block size; setting the position in the initial bitmask corresponding to at least one data block as a preset identifier to obtain the bitmask.

[0094] The preset data block size refers to the minimum data transmission granularity corresponding to the system-defined bitmask, such as 4 bytes.

[0095] Initial bitmask: refers to a bitmask template where all bits are 0 (or default values).

[0096] Preset identifier: refers to the symbol used to mark that a data block is requested, usually a binary "1".

[0097] For example, when the byte offset is 12, the data length is 8 bytes, and the preset data block size is 4 bytes, the calculation process is as follows:

[0098] Starting block index: 12 ÷ 4 = 3, indicating that the data starts in the 3rd data block;

[0099] End block index: (12+8-1)÷4=19÷4=4 (rounded down), indicating that the data covers the 4th data block;

[0100] At this point, set the 3rd and 4th bits of the initial bitmask (0000000000000000) to "1" and keep the remaining bits "0" to obtain the final bitmask: 0000000000011000.

[0101] Figure 5 A schematic diagram of a bitmask according to an embodiment of this application is shown.

[0102] like Figure 5 As shown, multiple threads of the application within the computing core (such as thread numbers 0x200, 0x10C, etc.) initiate initial memory access requests. These initial memory access requests first enter the cache line merging unit, which merges multiple initial memory access requests pointing to the same cache line, generating corresponding cache line addresses 510 (such as 0A100, 0A200, etc.). Simultaneously, the bitmask generation unit generates a 16-bit bitmask 520 for each merged cache line address 510 (e.g., the bitmask for 0A100 is 1010000100000000), used to identify the specific data block requested for access within that cache line. Finally, the memory access request consisting of "cache line address + bitmask" is output, achieving fine-grained data access encoding.

[0103] According to embodiments of this application, one or more data blocks are calculated based on byte offset, data length, and data block size. Then, in an initial bitmask (e.g., all zeros), the positions corresponding to these data blocks are set to preset identifiers (e.g., 1), generating the final bitmask. This achieves the mapping of byte-level access requests to data block-level bitmasks, realizing the conversion from coarse-grained addresses to fine-grained data identifiers, laying the technical foundation for subsequent on-demand, precise data transmission.

[0104] In some embodiments, the cache line merging unit is also used to perform a mask merging operation, specifically including: when multiple memory access requests initiated by any computing core point to the target cache line address, performing a mask merging operation on the bitmasks of the multiple memory access requests to generate a merged bitmask; generating a merge request based on the target cache line address and the merged bitmask, and sending the merge request to the target device.

[0105] The mask merging operation can be performed on multiple bitmasks by performing a bitwise OR operation to generate a bitmask containing information about all requested data blocks.

[0106] For example, when multiple threads' memory access requests point to the same cache line (such as thread 2 accessing address 0x100 and thread 3 accessing address 0x104), the bitmask generation unit performs a bitwise OR operation on the bitmask carried by each thread's memory access request to generate a merged bitmask.

[0107] A specific example is shown below, where three threads on the same computing core all access the same cache line 0x100:

[0108] If thread 1 accesses 0x101, the corresponding data block index is 0, then the bitmask of the memory access request for thread 1 is 0b000000000000001.

[0109] If thread 2 accesses 0x105, the corresponding data block index is 1, then the bitmask of the memory access request for thread 2 is 0b0000000000000010.

[0110] When thread 3 accesses 0x10D, the corresponding data block index is 3. Therefore, the bitmask of the memory access request for thread 3 is 0b0000000000001000.

[0111] Based on this, the bitmasks of the above three memory access requests are bitwise ORed to obtain the merged bitmask as follows: 0b000000000000001|0b0000000000000010|0b0000000000001000=0b0000000000001011.

[0112] By merging multiple memory access requests pointing to the same cache line into a single request, the overall number of requests is effectively reduced, thereby helping to alleviate transmission congestion on interconnect links and reduce the overhead of protocol processing and resource scheduling.

[0113] The bitmask generation unit can accurately determine the required valid data through a masking mechanism, thereby enabling the transmission of only the necessary data instead of the entire cache line. For example, when a request is made to access the second and third 4-byte blocks of cache line 0x100, the bitmask is 0b0000000000001100. After receiving the request, the remote accelerator only needs to return these two data blocks (a total of 8 bytes) without transmitting the complete 64-byte cache line, thus improving bandwidth utilization by 7 times.

[0114] After the cache line merging unit processes the memory access request, the system queries the L1 cache based on the memory access request. To avoid invalid cache operations, especially for cache-insensitive workloads (whose L1 cache hit rate is typically below 20%), directly writing remote data to the cache could lead to the eviction of valid local data, thus increasing access latency. Therefore, a cache bypass unit is introduced into the L1 cache to... Figure 6 The roadbed shown implements a buffer bypass function.

[0115] Figure 6 A schematic diagram illustrating the implementation logic of a cache bypass unit according to an embodiment of this application is shown.

[0116] like Figure 6 As shown, the cache bypass unit 600 maintains the physical address range (start address start_addr and end address end_addr) of the local accelerator memory. When any computing core requests memory access to query the L1 cache, the cache bypass unit 600 performs address verification on the target cache line address in the memory access request based on the physical address range of the accelerator to which the computing core belongs, and obtains the address verification result. If the address verification result indicates that the target cache line address is a remote address, a memory access request is sent to the target device. If the address verification result indicates that the target cache line address is a local address, the cache is accessed according to the traditional process.

[0117] Address verification of the target cache line address can include verifying whether the address of the target cache line is within the physical address range of the local accelerator.

[0118] By using the address verification mechanism of the cache bypass unit, remote memory access requests can bypass the local cache (especially for cache-insensitive workloads), avoiding invalid cache writes for cache-insensitive workloads, reducing pipeline blockage caused by cache conflicts, and improving overall cache efficiency.

[0119] According to an embodiment of this application, the system is provided with a cached request missing status holding register for storing memory access requests that are being processed (i.e., have been issued but have not yet received a response), for example, organized in the form of a list of pending requests.

[0120] Based on this, the above method may further include: matching the memory access request with pending requests in the pending request list before sending the memory access request to the target device to which the target cache line address belongs; and adding the memory access request to the associated queue of the target pending request in the pending request list in response to the existence of a target pending request requesting accessed data, without re-initiating remote access.

[0121] An associated queue is a queue that is associated with a pending request and is used to temporarily suspend new requests whose data is a subset of the data obtained by the pending request.

[0122] For example, when a memory access request arrives at the cache request missing status register, it is matched against the list of pending requests. If an entry pointing to the target cache line address exists in the list of pending requests, it means that a request has already been made for that target cache line address, and a missing status register hit occurs. The current memory access request will be recorded and await a response from the shared existing request (the target pending request). If no matching entry is found, a missing status register miss occurs, and the system will allocate a new register entry and send the request to the next lower memory hierarchy.

[0123] By maintaining a list of pending requests and matching requests, duplicate requests for the same or overlapping data blocks within the same cache line are avoided, improving the efficiency of request processing and reducing the number of remote accesses.

[0124] According to an embodiment of this application, matching a memory access request with pending requests in a pending request list includes: matching pending requests corresponding to cache line addresses that are consistent with the target cache line address from the pending requests to obtain candidate pending requests; and matching candidate pending requests whose bitmask coverage is greater than or equal to the bitmask coverage of the memory access request from the candidate pending requests to obtain the target pending request.

[0125] The bitmask coverage of the request refers to the set of data blocks corresponding to all bits set to "1" in the bitmask of the memory access request, that is, which data blocks the memory access request specifically requires.

[0126] The target request to be processed is a request selected from the candidate requests to be processed, whose bitmask coverage can completely cover the data required by the memory access request. In other words, the bits that are "1" in the bitmask of the target request to be processed must include all the bits that are "1" in the bitmask of the memory access request.

[0127] A candidate request is only ultimately identified as the target request if its bitmask coverage is greater than or equal to (i.e., a superset of) the coverage of a memory access request. This means that the request currently being processed has already (or will) obtained all the data required by the new request, and possibly more.

[0128] Figure 7 A schematic diagram illustrating request matching using a cached request missing status holding register according to an embodiment of this application is shown.

[0129] like Figure 7 As shown, the cache request missing status holding register maintains multiple cache line encoding entries (such as 0F002, 0X001, etc.). Each entry contains a cache address, a flag bit, data, and a bitmask. When a memory access request is input to the register, the address comparison unit in the cache request missing status holding register matches the target cache line address in the memory access request with the cache addresses of each cache line encoding entry. If a match is successful (for example, the target cache line address matches the cache address of entry 0F002), it is determined that an address hit has occurred. Next, bitmask matching is performed to determine whether the bitmask in the memory access request is a subset of the bitmask of the hit cache line encoding entry (such as entry 0F002). If a match is successful, a hit is output (such as entry 0F002), and the memory access request is associated with entry 0F002.

[0130] The matching results obtained by matching memory access requests with pending requests in the pending request list mainly include the following three cases: both address and bitmask match; address matches but bitmask does not match; and neither address nor bitmask matches. The following will combine... Figures 8-10 The above three situations will be explained in detail.

[0131] Figure 8 A schematic diagram is shown where the address and bitmask match according to an embodiment of this application.

[0132] When a memory access request arrives at the cache request missing status register, if there is a cache line encoding entry in the register whose address is the same as the target cache line address in the memory access request, and whose bitmask contains the bitmask of the memory access request (i.e., the bitmask of the memory access request is a subset of the bitmask of this entry), then it is determined that a missing status register hit has occurred.

[0133] like Figure 8 As shown, the target cache address of the memory access request is 0F100, and the bitmask is 0b01000000. A cache line encoding entry with address 0F100 and bitmask 0b01000000 already exists in the register. The bitmask of the memory access request is the same as the bitmask of this cache line encoding entry (a subset case), indicating that the fine-grained data obtained by the request being processed can meet the requirements of the memory access request. Therefore, it is determined that the register entry has been hit. At this time, the counter value of the flag bit of cache line encoding entry 0F100 is incremented by one. It should be noted that the flag bit in the cache line encoding entry is used to record the total number of requests associated with that entry.

[0134] The following example further illustrates the processing logic where both the address and bitmask match:

[0135] Prerequisite: A cache line encoding entry X exists in the holding register, with address 0x100 and bitmask 0b0000000000000011 (i.e., the 0th and 1st 4-byte blocks are being requested), and the request is not yet complete;

[0136] New request: cache line address 0x100, bitmask 0b0000000000000001 (request only block 0);

[0137] The register determination process is as follows:

[0138] Address matching: The new request matches the address of cache line encoded entry x, both being 0x100;

[0139] Bitmask matching: The bitmask of the new request (0b000000000000001) is a subset of the bitmask of cache line code entry X (0b0000000000000011) (all the "1" bits of the new request are present in cache line code entry X).

[0140] Processing result: A register hit is determined. The new request is added to the wait queue of cache line code entry X, and no new remote request is initiated. Once the request for cache line code entry X is completed, the new request can directly retrieve the data for block 0.

[0141] Figure 9 A schematic diagram is shown where the address matches but the bitmask does not match according to an embodiment of this application.

[0142] When a memory access request arrives at the cache request missing status register, if there is a cache line encoding entry in the register whose address is the same as the target cache line address in the memory access request, but whose bitmask does not contain all the bitmasks of the memory access request, then it is determined that a missing status register miss has occurred.

[0143] like Figure 9 As shown, the target cache address of the memory access request is 0F100, with a bitmask of 0b00000001. However, a cache line encoding entry already exists in the register at address 0F100 with a bitmask of 0b01000000. The bitmask of the memory access request differs from that of the cache line encoding entry, indicating that the fine-grained data being processed meets the requirements of the memory access request. Therefore, it is determined that a register entry miss has occurred. In this case, the system allocates an additional missing state holding register entry and sends the memory access request to the next lower memory hierarchy.

[0144] The following example further illustrates the handling logic for address matching but bitmask mismatch:

[0145] Prerequisite: A cache line encoding entry Y, address 0x100, bitmask 0b10000000000000000 (requesting only the 15th data block), already exists in the holding register; request incomplete.

[0146] New request: cache line address 0x100, bitmask 0b0000000000000001 (request only the 0th data block);

[0147] The register determination process is as follows:

[0148] Address matching: The new request matches the address of cache line encoded entry Y, both being 0x100;

[0149] Bitmask matching: The bitmask of the new request (0b0000000000000010) has no intersection with the bitmask of cache line encoded entry Y (0b10000000000000000) (all the "1" bits of the new request are not present in cache line encoded entry Y).

[0150] Processing result: The problem is determined to be a missing register. A new cache line encoding entry Z is allocated, with address 0x100 and bitmask 0b0000000000000010 recorded. A new request is then sent to the target device (requesting only the first data block).

[0151] Figure 10 A schematic diagram is shown where the address and bitmask do not match according to an embodiment of this application.

[0152] When a memory access request arrives at the cache request missing status register, if there is no cache line code entry in the register that matches the target cache line address in the memory access request, it is determined that a missing status register miss has occurred.

[0153] like Figure 10As shown, the target cache address for the memory access request is 0F200, with a bitmask of 0b00000000. The register only contains a cache line encoding entry at address 0F100 with a bitmask of 0b01000000; there is no cache line encoding entry matching 0F200. Therefore, the system allocates a new missing state-keeping register entry (e.g., the 0F200 entry) for the memory access request and forwards the memory access request to the next lower memory hierarchy.

[0154] The following example further illustrates the handling logic when both the address and bitmask do not match:

[0155] Prerequisite: A cache line encoding entry Z, address 0x100, bitmask 0b0000000000000010, already exists in the register; the request is incomplete.

[0156] New request: cache line address 0x200 (another cache line), bitmask 0b0000000000000001;

[0157] The register determination process is as follows:

[0158] Address mismatch (0x200 ≠ 0x100);

[0159] Processing result: Keep the register missing, allocate a new cache line encoding entry W, record address 0x200 and bitmask 0b0000000000000001, and initiate a remote request.

[0160] According to embodiments of this application, candidate requests for the same cache line are first selected based on address matching. Then, all data blocks required for the current request are matched from the candidate requests based on the bitmask coverage range. This mechanism, while ensuring the accuracy and reliability of request merging, avoids redundant remote memory accesses. It effectively reduces the number of requests and link load while achieving fine-grained data transmission, thus improving the overall data access efficiency of the multi-accelerator system.

[0161] The memory access requests in this application implement fine-grained requests. Although merging fine-grained transmissions reduces the amount of data in a single request, it increases the number of requests and may exacerbate link congestion.

[0162] Based on this, the accelerator of this application adds a remote direct memory access module, which realizes request traffic compression through remote encoding unit and remote decoding unit.

[0163] According to embodiments of this application, the target device includes at least one of the following: other computing cores, other accelerators, wherein the other computing cores are computing cores other than any computing core in the target accelerator to which any computing core belongs, and the other accelerators are accelerators other than the target accelerator in a multi-accelerator system.

[0164] Based on this, when the target device is another accelerator, sending a memory access request includes: sending a memory access request to the remote direct memory access module of the target accelerator, so that the remote direct memory access module adds the memory access request to the buffer queue of the target device; and when the buffer queue meets the request sending conditions, merging the requests in the buffer queue into a data packet and sending the data packet to the target device.

[0165] The buffer queue refers to the queue allocated to each accelerator in the remote direct memory access module for temporarily storing requests to be sent.

[0166] By performing request merging in the remote direct memory access module, fine-grained requests are further aggregated at the system level, reducing the number of data packets on cross-accelerator interconnect links, improving link bandwidth utilization efficiency, and reducing transmission latency.

[0167] According to an embodiment of this application, the request sending conditions include at least one of the following: the amount of data to be requested in the buffer queue is greater than or equal to a preset data amount threshold, and the number of cache lines to be requested in the buffer queue is greater than or equal to a preset number of cache lines.

[0168] Preset data volume threshold: The minimum amount of data required to trigger a request to be sent, such as 64 bytes.

[0169] Preset cache line count: The number of different cache lines required to trigger a request.

[0170] By setting reasonable request sending conditions, the number of requests is reduced while avoiding excessive delays caused by waiting for merging, thus achieving a good balance between bandwidth utilization and access latency.

[0171] The following is combined Figure 11 The remote direct memory access module is described.

[0172] Figure 11 A schematic diagram of a remote direct memory access module according to an embodiment of this application is shown.

[0173] like Figure 11 As shown, the remote direct memory access module 1100 includes a remote encoding unit 1110 and a remote decoding unit 1120.

[0174] The remote coding unit 1110 allocates an independent buffer queue to each accelerator. When it receives a local remote transmission request from a local accelerator, it temporarily stores the fine-grained requests destined for the same accelerator in the buffer queue. When the total amount of request data in the buffer queue reaches 64 bytes (or a timeout of 30 cycles), they are merged into a single data packet and sent to other accelerators via the internet system. For example, 10 4-byte requests can be merged into a 40-byte data packet (containing 10 2-byte response IDs), reducing the number of requests by 90%.

[0175] The remote decoding unit 1120 receives remote data responses from other accelerators through the Internet system, parses the "compression algorithm identifier" and "request quantity" fields in the header of the data packet, splits the merged data packet into independent fine-grained responses to obtain remote data, and distributes it to the corresponding computing core of the local accelerator.

[0176] The Remote Direct Memory Access (RDA) module achieves efficient fine-grained request aggregation and distribution through the collaborative work of the remote encoding and decoding units. The encoding unit maintains a buffer queue for each target accelerator, merging multiple fine-grained requests destined for the same target into a single data packet when capacity or time thresholds are reached, thus reducing the number of requests in the link. The decoding unit receives and parses these merged packets, breaking them down into independent fine-grained responses and accurately distributing them to the local computing core. This effectively alleviates congestion on interconnect links and reduces communication overhead while maintaining fine-grained data transmission capabilities, thereby contributing to improved overall throughput and efficiency of remote memory access in multi-accelerator systems.

[0177] To address the issue of unutilized high locality of cache-intensive workloads, a shared remote data cache (RDMA cache) is added before the remote direct memory access module. This RDMA cache is a dedicated shared cache for storing remote data. Located between the remote direct memory access module and local memory, the RDMA cache has a capacity of 1.5MB and can be implemented by "splitting the L2 cache" (e.g., allocating 0.5MB of the original 2MB L2 cache to the RDMA cache) without additional storage overhead.

[0178] RDMA caching allows different compute cores to reuse remote data in a shared architecture, overcoming the limitations of L1 cache's privacy. In addition, RDMA caching only stores remote data, avoiding competition with local data caches.

[0179] Based on this, when the target memory access strategy is a remote data caching strategy, the memory access method further includes: responding to the target cache line address in the memory access request initiated by any computing core being a remote address, querying the shared remote data cache according to the memory access request; responding to the shared remote data cache not hitting the data to be accessed, sending a memory access request to the target device, and caching the data obtained from the target device to the shared remote data cache.

[0180] According to embodiments of this application, for cache-intensive workloads, by introducing a shared remote data cache, the access locality of remote data is captured and utilized, transforming frequent remote accesses into fast local cache accesses, thereby helping to reduce the average access latency of workloads that rely on repetitive remote data and improving the overall system performance.

[0181] The RDMA cache in this application can efficiently organize and store non-contiguous valid small data blocks filtered by bitmasks into a dedicated RDMA cache for subsequent fast retrieval and reuse.

[0182] Figure 12 A schematic diagram of the process of remote direct memory access recoded storage according to an embodiment of this application is shown.

[0183] like Figure 12 As shown, firstly, a data extraction operation extracts multiple small data blocks (i.e., valid blocks marked by bitmasks) that are actually accessed from the complete cache line. Each small data block is accompanied by metadata such as identifier bits and type. Then, an address merging operation integrates multiple scattered small data blocks and their metadata into a single RDMA cache entry, which is finally stored in the RDMA cache, completing the RDMA recoding process. This mechanism enables efficient compressed storage and rapid reuse of remote fine-grained data.

[0184] Address merging operations can specifically include creating a new, compact cache entry structure for data blocks from the same original cache line. This structure contains aggregated header information (such as identifier bits, type, status bits, etc.) and contiguous valid data blocks, thereby achieving efficient utilization of storage space and fast lookup.

[0185] The RDMA cache in this application filters out non-contiguous valid data blocks using bitmasks and efficiently organizes and stores them in a dedicated cache structure to support subsequent fast retrieval and reuse. This helps to efficiently compress and store remote fine-grained data and quickly query it, thereby improving cache space utilization and data access efficiency.

[0186] In addition, to ensure compatibility with existing accelerator memory consistency protocols, the RDMA cache automatically invalidates all cache entries and writes any modified dirty data back to the corresponding remote memory when kernel execution ends (synchronization point). This mechanism ensures the consistency of cross-device memory data during parallel execution of multiple accelerators.

[0187] In summary, the memory access method provided in this application first intelligently classifies workloads through a decision engine, and then implements differentiated optimization: for cache-insensitive workloads, a "fine-grained transmission and request merging" mechanism is adopted to reduce the invalid bandwidth occupation of interconnect links; for cache-intensive workloads, a "dedicated remote cache" design is used to fully utilize the locality of remote data access while avoiding conflicts with local cache resources, ensuring that the performance benefits introduced by caching outweigh its overhead. This collaborative architecture of "automated decision engine + differentiated optimization mechanism" effectively improves the overall efficiency of remote memory access in multi-accelerator systems.

[0188] Based on the above memory access method, this application also provides a memory access device. The following will be combined with... Figure 13 The device is described in detail.

[0189] Figure 13 A structural block diagram of a memory access device according to an embodiment of this application is shown.

[0190] like Figure 13 As shown, the memory access device 1300 of this embodiment is applied to a multi-accelerator system and is executed by any computing core in the multi-accelerator system. The memory access device 1300 includes a first determining module 1310, a second determining module 1320, a generating module 1330 and a first sending module 1340.

[0191] The first determining module 1310 is used to determine a target memory access strategy from a preset memory access strategy set based on the local memory access characteristics of any computing core.

[0192] The second determining module 1320 is used to determine the location information of the data to be accessed in the target cache line based on the target cache line address in the initial memory access request initiated by any computing core when the target memory access strategy is a transmission optimization strategy.

[0193] The generation module 1330 is used to generate a bitmask for identifying the data to be accessed based on the location information. The bitmask represents the data block information occupied by the data to be accessed in the multiple data blocks obtained by dividing the target cache line according to the preset data block size.

[0194] The first sending module 1340 is used to add a bitmask to the initial memory access request to form a memory access request, and send the memory access request to the target device to which the target cache line address belongs. The target device returns the data to be accessed based on the bitmask in the memory access request.

[0195] According to an embodiment of this application, the location information includes the byte offset of the data to be accessed in the target cache line and the data length.

[0196] According to an embodiment of this application, the generation module includes: a first determining submodule and a setting submodule.

[0197] The first determining submodule is used to determine at least one data block in the target cache line corresponding to the data to be accessed, based on the byte offset, data length, and preset data block size.

[0198] The setting submodule is used to set the position corresponding to at least one data block in the initial bitmask as a preset identifier to obtain the bitmask.

[0199] According to an embodiment of this application, the above-mentioned apparatus further includes: a mask merging module and a second sending module.

[0200] The mask merging module is used to perform a mask merging operation on the bitmasks of multiple memory access requests when multiple memory access requests initiated by any computing core point to the target cache line address, and generate a merged bitmask.

[0201] The second sending module is used to generate a merge request based on the target cache line address and the merge bitmask, and send the merge request to the target device.

[0202] According to embodiments of this application, each computing core has a list of pending requests.

[0203] According to embodiments of this application, the above-described apparatus further includes a matching module and an adding module.

[0204] The matching module is used to match the memory access request with the pending requests in the pending request list before sending the memory access request to the target device to which the target cache line address belongs.

[0205] The Add module is used to add the memory access request to the associated queue of the target pending request in response to a target pending request that has a request for accessed data in the pending request list.

[0206] According to an embodiment of this application, the matching module includes: a first matching submodule and a second matching submodule.

[0207] The first matching submodule is used to match the pending requests with cache line addresses that match the target cache line address from the pending requests, and obtain candidate pending requests.

[0208] The second matching submodule is used to match candidate requests whose bitmask coverage is greater than or equal to the bitmask coverage of the memory access request from the candidate requests to be processed, so as to obtain the target request to be processed.

[0209] According to embodiments of this application, the target device includes at least one of the following: other computing cores, other accelerators, wherein the other computing cores are computing cores other than any computing core in the target accelerator to which any computing core belongs, and the other accelerators are accelerators other than the target accelerator in a multi-accelerator system.

[0210] According to an embodiment of this application, the first sending module includes a sending submodule.

[0211] The sending submodule is used to send memory access requests to the remote direct memory access module of the target accelerator when the target device is another accelerator, so that the remote direct memory access module adds the memory access requests to the buffer queue of the target device; when the buffer queue meets the request sending conditions, the requests in the buffer queue are merged into a data packet and sent to the target device.

[0212] According to an embodiment of this application, the request sending conditions include at least one of the following: the amount of data to be requested in the buffer queue is greater than or equal to a preset data amount threshold, and the number of cache lines to be requested in the buffer queue is greater than or equal to a preset number of cache lines.

[0213] According to embodiments of this application, the above-described apparatus further includes an address verification module and a third sending module.

[0214] The address verification module is used to verify the address of the target cache line before accessing the local cache of any computing core, based on the physical address range of the accelerator to which any computing core belongs.

[0215] The third sending module is used to send a memory access request to the target device in response to the address verification result indicating that the target cache line address is a remote address.

[0216] According to an embodiment of this application, the above-described apparatus further includes a query module and a fourth sending module.

[0217] The query module is used to query the shared remote data cache in response to a memory access request initiated by any computing core, where the target cache line address is a remote address, when the target memory access policy is a remote data caching policy.

[0218] The fourth sending module is used to send a memory access request to the target device in response to a missed data in the shared remote data cache, and to cache the data obtained from the target device to the shared remote data cache.

[0219] According to embodiments of this application, any plurality of modules among the first determining module 1310, the second determining module 1320, the generating module 1330, and the first transmitting module 1340 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the first determining module 1310, the second determining module 1320, the generating module 1330, and the first transmitting module 1340 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first determining module 1310, the second determining module 1320, the generating module 1330, and the first sending module 1340 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0220] Figure 14 A block diagram of an electronic device suitable for implementing a memory access method according to an embodiment of this application is shown.

[0221] like Figure 14 As shown, an electronic device 1400 according to an embodiment of this application includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage portion 1408 into a random access memory (RAM) 1403. The processor 1401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1401 may also include onboard memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0222] RAM 1403 stores various programs and data required for the operation of electronic device 1400. Processor 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Processor 1401 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 1402 and / or RAM 1403. It should be noted that the programs may also be stored in one or more memories other than ROM 1402 and RAM 1403. Processor 1401 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0223] According to embodiments of this application, the electronic device 1400 may further include an input / output (I / O) interface 1405, which is also connected to a bus 1404. The electronic device 1400 may also include one or more of the following components connected to the input / output (I / O) interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output (I / O) interface 1405 as needed. A removable medium 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1410 as needed so that computer programs read from it can be installed into the storage section 1408 as needed.

[0224] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0225] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 1402 and / or RAM 1403 and / or one or more memories other than ROM 1402 and RAM 1403 described above.

[0226] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0227] When the computer program is executed by the processor 1401, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0228] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1409, and / or installed from the removable medium 1411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0229] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1409, and / or installed from the removable medium 1411. When the computer program is executed by the processor 1401, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0230] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0231] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0232] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0233] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A memory access method applied to a multi-accelerator system, comprising: The method is executed by any computing core in the multi-accelerator system, and the method comprises: According to the local memory access characteristics of the any computing core, a target memory access strategy is determined from a preset memory access strategy set; In the case that the target memory access strategy is a transmission optimization strategy, the position information of the to-be-accessed data in a target cache line is determined according to the target cache line address in the initial memory access request initiated by the any computing core, wherein the position information comprises a byte offset and a data length of the to-be-accessed data in the target cache line; According to the byte offset and the data length, and a preset data block size, at least one data block corresponding to the to-be-accessed data in the target cache line is determined; The positions corresponding to the at least one data block in the initial bit mask are set as a preset identifier to obtain a bit mask, wherein the bit mask represents the data block information occupied by the to-be-accessed data from a plurality of data blocks obtained by dividing the target cache line according to the preset data block size; The bit mask is added to the initial memory access request to form a memory access request; In the case that a plurality of memory access requests initiated by any computing core point to a target cache line address, a bitwise OR operation is performed on the bit masks of a plurality of the memory access requests to generate a merged bit mask; According to the target cache line address and the merged bit mask, a merged request is generated, and the merged request is sent to the target device, and the target device returns the to-be-accessed data according to the bit mask in the merged request.

2. The method of claim 1, wherein, The any computing core has a to-be-processed request list; The method further comprises: Before sending the memory access request to a target device to which the target cache line address belongs, the memory access request is matched with a to-be-processed request in the to-be-processed request list; In response to the existence of a target to-be-processed request requesting the to-be-accessed data in the to-be-processed request list, the memory access request is added to an associated queue of the target to-be-processed request.

3. The method of claim 2, wherein, The matching of the memory access request with the to-be-processed request in the to-be-processed request list comprises: According to the target cache line address, a to-be-processed request corresponding to a cache line address consistent with the target cache line address is matched from the to-be-processed request to obtain a candidate to-be-processed request; According to the bit mask coverage range of the memory access request, a candidate to-be-processed request with a bit mask coverage range greater than or equal to the bit mask coverage range in the memory access request is matched from the candidate to-be-processed request to obtain the target to-be-processed request.

4. The method of claim 1, wherein, The target device comprises at least one of the following: other computing cores, other accelerators, wherein the other computing cores are computing cores other than the any computing core in a target accelerator to which the any computing core belongs, and the other accelerators are accelerators other than the target accelerator in the multi-accelerator system; In the case that the target device is the other accelerator, the sending of the memory access request comprises: sending the memory access request to a remote direct memory access module of the target accelerator, so that the remote direct memory access module adds the memory access request to a buffer queue of the target device; and in a case where the buffer queue satisfies a request sending condition, merging requests in the buffer queue into a data packet and sending the data packet to the target device.

5. The method of claim 4, wherein, The request sending condition includes at least one of the following: a data amount to be requested in the buffer queue is greater than or equal to a preset data amount threshold, and a cache line amount to be requested in the buffer queue is greater than or equal to a preset cache line amount.

6. The method of claim 1, wherein, Further comprising: before accessing a local cache of the any computing core, performing address verification on the target cache line address according to a physical address range of an accelerator to which the any computing core belongs; in response to an address verification result indicating that the target cache line address is a remote address, sending the memory access request to the target device.

7. The method of claim 1, wherein, In a case where the target memory access strategy is a remote data cache strategy, the method further comprises: in response to a target cache line address in the memory access request initiated by the any computing core being a remote address, querying a shared remote data cache according to the memory access request; in response to the shared remote data cache not hitting the data to be accessed, sending the memory access request to the target device and caching data obtained from the target device to the shared remote data cache. 8.An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-7.

Images