GPGPU cross-SM data sharing method in multi-Kernel scene

Abstract

The invention provides a GPGPU cross-SM data sharing method in a multi-Kernel scene, and belongs to the technical field of GPGPU parallel computing and cache optimizing.The method comprises the steps that an access counter and a resident identifier are additionally arranged for L1D Cache line data of each SM in a GPGPU, the access counters are used for counting the access frequency of the L1D Cache line data in a set time period, and the resident identifier is stored in the L1D Cache line data in the set time period; data accessed by high frequency in each SM are accurately screened as high-frequency shared data, and then the high-frequency shared data are resided through the residing identifier, so that the high-frequency shared data are prevented from being eliminated by a traditional cache replacement strategy, and meanwhile, the unique identifier of the data and the information of the SM to which the data belong are recorded. According to the method, the high-frequency shared data is accurately positioned and solidified, so that a foundation is laid for subsequently constructing an efficient cross-SM data sharing link, a data access path is effectively shortened, cache resource waste is reduced, and the overall data processing efficiency and throughput performance in a multi-kernel concurrent scene are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of GPGPU parallel computing and cache optimization technology, specifically relating to a method for GPGPU cross-SM data sharing in a multi-Kernel scenario. Background Technology

[0002] General-purpose graphics processing units (GPGPUs), with their high parallel computing capabilities, are widely used in scientific computing, cloud inference services, deep learning training, and graphics rendering, and are primarily responsible for executing high-throughput parallel tasks. With the rapid growth in demand for large-scale parallel computing platforms, especially in high-concurrency scenarios such as image processing, large model inference, and real-time simulation computing, concurrent execution of multiple kernels has become the mainstream approach to improving task processing efficiency. This places even more stringent requirements on the efficient memory access mechanisms of GPGPUs.

[0003] In GPU parallel computing, a kernel typically refers to a parallel computing task or program executed on the GPU. In multi-kernel scenarios, the GPGPU executes multiple computing tasks (kernels) in parallel through its internal multiple streaming multiprocessors (SMs). However, the existing GPGPU hardware architecture has inherent limitations, leading to significant memory access performance bottlenecks in multi-kernel concurrent scenarios. When multiple kernels are running concurrently, a large number of threads compete for access to the limited cache resources of the SM, easily causing frequent cache replacements and a significant increase in L1D cache miss rates. Ultimately, this results in a substantial increase in memory access latency, hindering the overall performance improvement of the GPGPU.

[0004] To alleviate these problems, existing research has proposed various cache optimization schemes, such as the locality-based last-level cache (LLC) design and shared data identification methods based on access pattern prediction. However, these schemes are difficult to adapt to the actual needs of multi-kernel concurrent scenarios. Their core designs are all geared towards single-kernel scenarios and lack the ability to adapt to the differences in memory access characteristics between multiple kernels. They cannot cope with the complex SM resource contention and data interaction requirements of GPGPUs across multiple kernels. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for data sharing across SMs in a multi-Kernel scenario using GPGPU.

[0006] To achieve the above objectives, this invention provides a method for GPGPU data sharing across SMs in a multi-Kernel scenario, comprising: In a multi-kernel scenario, an access counter and a residency identifier are added to the L1D Cache line data of each streaming multiprocessor (SM) in the target general-purpose graphics processing unit (GPGPU). The access counter counts the access frequency of the L1D Cache line data within a set time period. Based on the access frequency, the L1D Cache line data that is accessed frequently in each streaming multiprocessor is selected and recorded as high-frequency shared data. A residency operation is performed on the high-frequency shared data according to the residency identifier, and the unique identifier of the high-frequency shared data and its corresponding streaming multiprocessor information are recorded.

[0007] A cross-SM data sharing table is designed for the target GPGPU. The cross-SM data sharing table includes: a unique identifier for high-frequency shared data, the number of cross-cell accesses of high-frequency shared data, and a valid bit. The valid bit is used to mark the availability status of the corresponding high-frequency shared data.

[0008] When GPGPU executes concurrently, the kernel corresponding to each SM and the storage address of each SM in the cross-SM data sharing table are obtained, and a global mapping table is constructed. When a certain SM has missing computation data, the global mapping table is used to query whether the data is in other SMs. If the target computation data is found and its valid bit is marked as available, the target computation data is directly obtained from the corresponding SM to achieve data sharing.

[0009] Preferably, when a certain SM has missing computational data, the method further includes: if the target computational data is not found, it is obtained from the lower-level storage unit through the traditional data access path; when the cross-SM data sharing table reaches the storage limit, the resident identifier is set to the unlocked state and the high-frequency shared data with the fewest cross-unit accesses is deleted; when the kernel execution of the target general-purpose graphics processor ends, the cross-SM data sharing table is cleared.

[0010] Preferably, the access counter is configured in the L1D Cache line, specifically including: the access counter is a dual-time access counter, divided into a first-half access counter FC and a second-half access counter BC; FC is used to count the access frequency of L1D Cache line data in the first half of a set sampling period, and BC is used to count the access frequency in the second half of the same sampling period; only when the frequencies counted by FC and BC both reach a preset high-frequency threshold, the corresponding L1D Cache line data is selected as high-frequency shared data.

[0011] Preferably, when a certain SM has missing computational data, the global mapping table is used to query whether the data is in other SMs. If the target computational data is found and its valid bit is marked as available, the target computational data is directly obtained from the corresponding SM to achieve data sharing. Specifically, when a certain SM has missing computational data, the SM is recorded as a data request unit. The data request unit locates the SM to which the target missing data belongs through the global mapping table and sends a data access request to the target SM. The target SM verifies the residence status and valid bit of the high-frequency shared data in its own L1D Cache. If the status is valid, the target SM directly reads the data from its own L1D Cache and transmits it to the request unit, while accumulating the cross-unit access count of the data in the cross-SM data sharing table.

[0012] Preferably, the valid bit of the cross-SM data sharing table is identified by binary, and the specific rules include: when the valid bit is 1, it indicates that the high-frequency shared data is available and the corresponding L1D Cache line is in a resident state; when the valid bit is 0, it indicates that the high-frequency shared data is unavailable and the corresponding L1D Cache line is in an unlocked state; the valid bit state and the resident identifier state are linked in real time, and the state switch is updated synchronously.

[0013] Preferably, the global mapping table specifically includes: a kernel identifier field, used to record the kernel numbers of each kernel executed concurrently by the general-purpose graphics processor; a streaming multiprocessor (SM) identifier field, used to associate the SM number corresponding to each kernel; and a storage address field for the cross-SM data sharing table, used to record the physical address of the corresponding SM cross-SM data sharing table in on-chip storage; the kernel identifier field allows for quick indexing of the corresponding SM and cross-SM data sharing table, shortening query time.

[0014] Preferably, when the cross-SM data sharing table reaches its storage limit, setting the resident identifier to the unlocked state and deleting the high-frequency shared data with the fewest cross-cell accesses specifically includes: traversing the data entries of the current SM in the cross-SM data sharing table, filtering out one or more data entries with the fewest cross-SM data accesses; verifying whether the L1DCache row data corresponding to the filtered entry is in an unaccessed state; for unaccessed data, first setting the resident identifier to the unlocked state and the valid bit to 0, then deleting the entry from the cross-SM data sharing table, and simultaneously releasing the corresponding storage space of the L1D Cache.

[0015] Preferably, when the kernel execution of the target general-purpose graphics processor ends, clearing the cross-SM data sharing table specifically includes: clearing all entries of the current SM in the cross-SM data sharing table, resetting the storage pointer of the cross-SM data sharing table; setting the resident identifiers corresponding to all L1D Cache lines that were previously resident in the SM to the unlocked state; and uniformly updating the valid bits of all associated high-frequency shared data to 0 to ensure that no invalid resident data remains.

[0016] Preferably, the L1D Cache row data further includes a row address field, which is used to store the physical address of the L1D Cache corresponding to the frequently shared data. When it is necessary to unlock or delete data, the target L1D Cache row can be directly located through this address field. When accessing data across SMs, the target SM can quickly read the data in the L1D Cache through this address without traversing the cache.

[0017] This invention also provides a system for a GPGPU cross-SM data sharing method in a multi-Kernel scenario, comprising: The L1D cache module is used to add access counters and residency identifiers to the L1D Cache lines of data in each streaming multiprocessor (SM) of the target general-purpose graphics processing unit (GPGPU) in a multi-kernel scenario. The access counters count the access frequency of the L1D Cache lines within a set time period. Based on the access frequency, the frequently accessed L1D Cache lines in each streaming multiprocessor are selected and recorded as high-frequency shared data. Residency operations are performed on these high-frequency shared data according to the residency identifiers, and the unique identifier of the high-frequency shared data and its associated streaming multiprocessor information are recorded.

[0018] The shared tag table module is used to design a cross-SM data sharing table for the target GPGPU. The cross-SM data sharing table includes: a unique identifier for high-frequency shared data, the number of cross-cell accesses of high-frequency shared data, and a valid bit. The valid bit is used to mark the availability status of the corresponding high-frequency shared data.

[0019] The Kernel and SM mapping table module is used to obtain the kernel corresponding to each SM during concurrent GPGPU execution and the storage address of each SM in the cross-SM data sharing table, and construct a global mapping table. When a certain SM has missing computation data, the global mapping table is used to query whether the data is in other SMs. If the target computation data is found and its valid bit is marked as available, the target computation data is directly obtained from the corresponding SM to achieve data sharing.

[0020] The data sharing method for GPGPU across SMs in a multi-Kernel scenario provided by this invention has the following beneficial effects: This invention adds a counter and residency mechanism to the L1D Cache line to accurately filter and reside high-frequency shared data. Combined with a shared tag table and a Kernel-SM mapping table, it constructs a direct communication link between SMs. This overcomes the latency disadvantage caused by traditional GPUs where data sent by the SM can only be retrieved from the lower-level cache via the interconnect network when data transmission is missing. The data acquisition path is shortened from SM-interconnect network-lower-level cache to SM-SM, significantly reducing data access latency in multi-Kernel concurrent scenarios. Simultaneously, through dynamic management of the valid bit and access count, it ensures the accuracy of data sharing and efficient utilization of cache resources, significantly improving the overall execution performance of GPGPU in multi-task concurrency. Attached Figure Description

[0021] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a method for GPGPU data sharing across SMs in a multi-Kernel scenario according to an embodiment of the present invention. Figure 2 This is an architecture diagram of a GPGPU cross-SM data sharing method in a multi-Kernel scenario according to an embodiment of the present invention; Figure 3 This is a comparison chart of the system throughput of a GPGPU cross-SM data sharing method in a multi-Kernel scenario according to an embodiment of the present invention with other methods; Figure 4 This is a comparison chart of the average normalized turnaround time of a GPGPU cross-SM data sharing method in a multi-Kernel scenario according to an embodiment of the present invention with other methods; Figure 5 This is a power consumption comparison chart of a GPGPU cross-SM data sharing method in a multi-Kernel scenario according to an embodiment of the present invention and other methods. Detailed Implementation

[0023] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0024] This invention proposes a collaborative method based on an on-chip cross-SM data sharing table (CSDT) and a cache residency mechanism: At the hardware / analog level, a new KSMT (recording the mapping between SM and Kernel) and a dedicated STT (Shared Tag Table, with a fixed number of entries, e.g., 32 entries per SM) are added to record strongly reusable shared data. An access count field (FC / BC) and a residency lock bit L are added to the L1D cache line to identify blocks that are frequently hit in both the first and second halves within a fixed sampling period and write them to the CSDT, while simultaneously residing the corresponding cache line to indicate that it is not replaceable. When an L1D access failure occurs, a parallel query to the CSDT is initiated; if a hit occurs, data is returned directly from the corresponding SM through a new path to bypass the next-level cache; if a miss occurs, it falls back to the traditional path. To ensure dynamic resource availability, the STT adopts a replacement strategy based on the shared access count AC (replacing the smallest item in the AC and clearing the corresponding lock bit), and the allocated STT entries are released through KSMT at the end of the Kernel operation. This method effectively reduces L1D miss rate and memory access pressure by identifying and residing highly reusable shared data across SMs, providing low-latency cross-SM data return paths, and dynamic management based on AC, thereby improving memory access efficiency and energy efficiency in multi-Kernel concurrent environments.

[0025] This invention provides a method for GPGPU data sharing across SMs in multi-Kernel scenarios, specifically as follows: Figure 1 As shown, it includes: Step S1: Modify the original architecture of the on-chip cache to effectively develop data locality between SMs.

[0026] Figure 2 The logical structure diagram of CSDSR is shown. Figure 2The upper part is the CSDT (Custom Data Table). The CSDT's composition and internal structure specifically include: the Shared Data Label Table (CSDT), which comprises the Kernel-SM Mapping Table (KSMT) and the Shared Label Table (STT), used to maintain the correspondence between Kernels, SMs, and shareable L1D cache blocks in multi-Kernel concurrent execution scenarios. KSMT entries include the Kernel Identifier field (KID), the SM Identifier field (SID), and the Starting Index field (SP), representing the Kernel and SM numbers currently being executed by the SM, and the starting position of the SM's entry in the STT. The STT records L1D cache block information with cross-SM shared attributes. Its entries include the Cache Block Label field (Tag), the Access Count field (AC), and the Validity Flag field (V). The Tag identifies the L1D cache block, AC records the access count of the corresponding cache block, and V indicates whether the entry is valid. KSMT and STT establish a mapping relationship through the Starting Index field (SP), ensuring that each SM corresponds to a set of consecutive entries in the STT, thereby achieving ordered management of shared data labels.

[0027] Figure 2 The lower half is a diagram illustrating the collaborative relationship between the CSDT, SM, and L1D cache. Specifically, during GPU execution, the CPU allocates kernels to multiple SMs for parallel execution via the thread scheduler. Each SM includes registers, load memory units, an L1D cache, and a miss handling structure. When an SM misses an access to the L1D cache, the access request queries the CSDT via the data bus to determine if a shareable cache block exists and is held by another SM. If a valid entry matching the tag of the accessed cache block is found in the CSDT, the required data is retrieved via the on-chip interconnect network based on the SM number corresponding to that entry. The sharing flag in the L1D cache indicates whether the cache block participates in cross-SM sharing and is consistent with the record in the CSDT, thereby enabling data sharing and collaborative access among multiple SMs.

[0028] To effectively develop data locality between SMs and enable strongly reusable shared data between SMs to be persistently retained in the L1D cache, so that in the event of an L1D cache access failure, the data can be hit in the L1D cache of other SMs and transferred to the requesting SM through a newly established data path, thereby reducing the long-latency access overhead in the event of an L1D cache access failure, a Cross-SM Data Sharing Table (CSDT) was designed between SMs.

[0029] To save on the hardware overhead of the CSDT, it is designed to consist of a kernel-SM mapping table (Kernel SMMapping Table, KSMT) and a shared tag table (STT). The STT records basic information about strongly reusable shared data between SMs, including the tag field of the cache block corresponding to the strongly reusable shared data, the AC field representing the cumulative number of times the strongly reusable shared cache block was accessed by different SMs during the sampling period, and the valid bit V field for each entry in the STT.

[0030] KSMT records the mapping relationship between SM and Kernel. Each entry includes the SM number SID, the Kernel number KID running on each SM, and the starting entry number SP assigned to each SM in STT.

[0031] To support the execution of the method, this invention modifies the L1D cache. Specifically, FC and BC fields are added to the L1D cache line on each SM chip, representing the number of times a certain L1D cache line is accessed in the first and second halves of the sampling period, respectively. When the values ​​of FC and BC of an L1D cache block both exceed a set threshold, the L1D cache block is considered to have high reusability and shareability. The basic information of the L1D cache block is then recorded in the Shared Tag Table (STT), and the corresponding SM and Kernel are written to the KSMT table. Additionally, a field L is set in the L1D cache block to indicate whether the cache block resides in the L1D cache, to more effectively support data sharing between SMs. This field is stored using 1 bit; a value of 1 indicates that the corresponding L1D cache block cannot be replaced, and a value of 0 indicates that it can be replaced.

[0032] Step S2: When multiple kernels execute concurrently, allocate a different number of SM resources to each kernel according to the SM partitioning strategy, and then initialize the correspondence between SM and kernel in the KSMT table according to the SM allocation for STT query.

[0033] When multiple kernels execute concurrently, a different number of SM resources are allocated to each kernel according to the SM partitioning strategy. Then, the mapping between SMs and kernels is initialized in the KSMT table based on the SM allocation. Afterwards, statistical analysis is performed on L1D cache access during each sampling period. To achieve cross-SM data sharing, highly reusable shared L1D cache blocks are identified by analyzing L1D cache access patterns, and their information is saved in the CSDT. When an L1D cache access failure occurs in a certain SM, the STT is queried in the CSDT via KSMT. If an STT entry with the same tag as the required cache block exists, the corresponding SM is located using KSMT, and the required data is retrieved from it.

[0034] Step S3: Collect data through the sampling period and write it into the CSDT.

[0035] After the sampling period begins, the FC and BC fields of each L1D cache block are set to 0. When an L1D cache block is accessed and hit in the first half of the sampling period, the FC field value of that cache block is incremented by 1; if it is accessed and hit in the second half of the sampling period, the BC field value of that cache block is incremented by 1. When both the FC and BC values ​​exceed a certain threshold, the L1D cache is considered to have high reusability and shareability, and the information of that L1D cache is written to the CSDT.

[0036] Algorithm 1 defines the entire process of writing highly reusable shared L1D cache block information into the CSDT as follows: When a highly reusable shared L1D cache block is detected in a certain SM, the system obtains the tag value corresponding to the L1D cache block and its SM number, and writes the relevant information into the shared data tag table CSDT. Specifically, firstly, based on the SM number, the starting index position allocated to the SM in the shared tag table STT is determined in the Kernel-SM mapping table. Subsequently, within the range of STT entries corresponding to the starting index position, each STT entry is checked sequentially to determine whether the L1D cache block tag to be written already exists in the current STT entry. When it is confirmed that the L1D cache block tag does not exist, an invalid free entry is searched within the range of the STT entries, and the tag of the L1D cache block is written into the free entry. At the same time, the access count field of the entry is initialized, and its valid flag is set to valid. After the STT entry is written, the shared flag bit of the cache line corresponding to the L1D cache block tag in the SM is set to valid to indicate that the cache block has been marked as a data block that can be shared across SMs, thereby completing the update process of the Shared Data Tag Table (CSDT).

[0037] Algorithm 1 demonstrates writing the tag information of an L1D cache block determined to have high reusability and sharing into the CSDT. Here, L1D_cacheblock_tag and Smid represent the tag value and corresponding SM number of the L1D cache block with high reusability and sharing, respectively. The algorithm first uses the SMID to find the starting index SP allocated to the SM in the STT table. Then, it searches for a free entry in the corresponding SM's STT table and writes the tag to the found free entry. Simultaneously, it sets the L field of the corresponding SM's L1D cache block to 1, indicating that the cache line must reside in the L1D cache and cannot be replaced until the corresponding entry in the STT is replaced.

[0038] Step S4: When an L1D cache access of SM fails, it sends a memory access request to the next level cache while simultaneously initiating a query request to CSDT.

[0039] When an L1D cache access fails in a Service Memory Provider (SM), it sends a memory access request to the next lower-level cache while simultaneously initiating a query request to the Service Query Data Table (CSDT). At this point, each entry in the Service Memory Management Table (KSMT) is traversed sequentially, and the query begins at the corresponding position in the Service Query Table (STT) based on the SP value corresponding to each SM. If the tag information of the failed cache block is found in the STT entry for that SM, the corresponding SM number is returned to the SM requesting the CSDT query, and the corresponding cache block is retrieved from the SM with the found SM number and added to the SM requesting the CSDT query. Afterward, the corresponding memory access request in the Service Memory Registry (MSHR) of that SM is cancelled.

[0040] The steps for querying the Shared Data Tag Table (CSDT) are defined as Algorithm 2, specifically including: When an L1D cache access to a certain SM fails, the system queries the CSDT based on the tag value corresponding to the L1D cache block where the access failure occurred, to determine whether there is a data source that can be shared across SMs. Specifically, the system sequentially traverses each SM and, for each SM, accesses the CSDT entry allocated to that SM based on its starting index position recorded in the Kernel-SM mapping table. For each accessed CSDT entry, it is determined whether it is in a valid state, and further, whether the tag value recorded therein matches the tag of the L1D cache block where the access failure occurred. When an entry matching the L1D cache block tag and in a valid state is detected in a certain CSDT entry, the access count field of that entry is updated to reflect the situation where the shared data has been accessed again, and the corresponding SM number is returned as the query result. If no valid entry matching the L1D cache block label is found in any STT entry after the traversal is completed, it is determined that there is no shareable data source and a miss result is returned.

[0041] Algorithm 2 demonstrates the algorithm for querying the CSDT when an SM access fails. The outer loop iterates through the cache to obtain the starting entry position of each SM in the STT, while the inner loop iterates through each entry allocated to the corresponding SM in the STT. If the tag of the failed L1D cache block is found in the CSDT, it indicates that the failed access can be hit in another SM, and the corresponding SM number is returned directly. If it is not found in the STT, -1 is returned.

[0042] Step S5: Modify the residence and replacement of highly reusable shared data based on the sharing status shown by the L1D cache access status and the usage status of CSDT resources.

[0043] Algorithm 3 defines the replacement method for the Shared Tag Table (STT), which specifically includes: when a new shareable L1D cache block tag needs to be written to a certain SM and there is no free entry in its corresponding Shared Tag Table (STT), the system performs a replacement operation on the STT entries allocated to that SM. Specifically, the system first determines the starting position of the STT entries allocated to that SM in the Kernel-SM mapping table based on the SM number. Then, within the range of STT entries corresponding to the starting position, the access count field of each STT entry is compared to determine the STT entry with the smallest access count value. After determining the STT entry with the smallest access count value, the system resets the sharing flag bit of the L1D cache block corresponding to the tag recorded by that STT entry to release the sharing state of that cache block. Subsequently, the STT entry with the smallest access count value is replaced with the L1D cache block tag to be written, and the access count field of that entry is initialized, thereby completing the replacement and update of the STT entry.

[0044] In step S3, the present invention sets an L field for each SM's L1D cache block to retain highly reusable and shared cache blocks in the L1D cache, thereby enabling more efficient sharing of these cache blocks among different SMs. When it is determined that an L1D cache block in an SM has high reusability and sharing characteristics, the information of that L1D cache block is written into the CSDT, and the L value of that L1D cache line is set to 1, so that the cache block resides in the L1D cache. However, if a cache block resides for a long time, it will also crowd out the usage space of other cache blocks. Therefore, its residency attribute should be modified according to the memory access situation in step S4 and the usage of CSDT resources, which can be divided into the following two cases.

[0045] i. When the STT table is full, if new highly reusable shared data needs to be written, it needs to be replaced in the allocated STT table entries. The replacement of STT table entries depends on the AC field value of the entry, i.e., the number of times the highly reusable shared L1D cache row corresponding to the entry has been accessed by other SMs. The algorithm of this invention selects the entry with the smallest AC value for replacement. Algorithm 3 illustrates the STT table entry replacement process. The for loop selects the entry with the smallest AC value from all STT table entries allocated by the SM that needs to replace the entry. Then, it replaces the original entry's Tag value with the Tag of the newly inserted L1D cache block in the STT, sets its AC field to 0, and sets the L1D cache block resident bit L corresponding to the replaced entry to 0.

[0046] ii. When the Kernel finishes execution, its allocated CSDT resources are released, and the L value of all SMs corresponding to the Kernel in the STT table with the resident attribute in the L1D cache row is set to 0.

[0047] Step S6: When the concurrent kernel execution ends, the resources allocated by the kernel in CSDT need to be released, including the resources allocated in KSMT and STT.

[0048] Algorithm 4 defines the method for releasing shared data label table (CSDT) resources. Specifically, when a kernel finishes execution, the system releases the resources occupied by that kernel in the CSDT to reclaim the entries allocated in the Kernel-SM mapping table and the STT (Shared Tag Table). Specifically, the system traverses the Kernel-SM mapping table to find all SMs corresponding to the identifier of the completed kernel. For each SM associated with the kernel, the system locates the range of STT entries allocated to that SM based on its starting index position recorded in the Kernel-SM mapping table. Within the STT entry range, the system invalidates the valid flag position of each STT entry and simultaneously resets the shared flag bit of the L1D cache line corresponding to the tag recorded in the STT entry in the SM, thereby releasing the shared state of the corresponding cache block. Subsequently, the Kernel-SM mapping table entry corresponding to the SM is marked as invalid, thus completing the release process of the shared data label table resources associated with that kernel.

[0049] When a concurrent kernel finishes execution, the resources allocated by the kernel in the CSDT need to be released, including those allocated in KSMT and STT. For KSMT, simply set the KernelID field in each entry for that kernel to -1. For STT, set the valid V bit of all entries in STT for each SM allocated by that kernel to 0. Additionally, remove the residency attribute from the L1D cache lines residing in the SMs allocated by the kernel, i.e., set the L field of the corresponding L1D cache line to 0, restoring its normal access attributes. The specific process of releasing CSDT resources is shown in Algorithm 4. The outer loop analyzes each entry in KSMT line by line. If it corresponds to a kernel that has finished execution, it locates the corresponding position in STT based on the value of the SP field in the entry. The inner loop sets the V field of the 32 consecutive entries starting from the position pointed to by SP to 0 and sets the residency attribute in the L1D cache line corresponding to the STT entry to 0.

[0050] To ensure a fair and comprehensive evaluation of this invention, all experiments were conducted on GPGPU-Sim (v4.0.0), a periodic simulator that simulates the GPGPU architecture. The simulated GPGPU architecture was the Turing architecture corresponding to NVIDIA's RTX 2060 product, comprising 30 SMs and 30 clusters, with each cluster containing only one SM. Each SM also includes a 64KB L1D cache. Detailed GPGPU configuration information is shown in Table 1. Furthermore, the source code of the GPGPU-Sim simulator was modified to implement the proposed method CSDSR.

[0051] To verify the effectiveness of the CSDSR method in a multi-kernel environment, eight test programs were selected from test suites such as ISPASS-2009, NVIDIA CUDASDK, Rodinia, and PolyBench. These were then paired to construct 12 test program pairs. Table 2 shows the list of selected test programs.

[0052] The proposed method CSDSR optimizes on-chip L1 cache access efficiency during concurrent execution of multiple kernels based on GPGPU resource allocation. Therefore, this paper compares it with two SM partitioning strategies, Even and Dynamic Optimizations, and applies CSDSR to these two strategies (i.e., Even+CSDSR, Dynamic Optimizations+CSDSR), and then compares the performance before and after applying CSDSR. In the Even strategy, the same number of SMs are allocated to multiple concurrently executing kernels. The Dynamic Optimizations strategy classifies the kernels executing in parallel and performs static SM partitioning for different types of kernel combinations based on experience. This invention analyzes the system throughput (STP), average normalized turnaround time (ANTT), and power consumption. STP is a system-level evaluation index for multi-kernel execution performance, and its calculation is shown in formula (1). Indicates the number of kernels currently executing concurrently; subscript Indicates the first One kernel, ; Indicates the first The number of execution cycles required for each kernel in Single Program (SP) mode; Indicates the first The number of execution cycles required for a kernel in Multi-Program (MP) concurrent execution mode.

[0053] (1) Figure 3 The STP performance of the test program combinations under four strategies is demonstrated. Experimental results show that after adding the CSDSR mechanism, compared with the original EVEN strategy and the Dynamic Optimizations strategy, the average STP of EVEN+CSDSR and Dynamic Optimizations+CSDSR is improved by approximately 12.37% and 11.26%, respectively. This is due to the fact that the CSDSR method effectively develops data locality between SMs, which has a good effect on improving the overall data reusability of the system and reducing on-chip memory access conflicts. It also shows that CSDSR is effective for different SM resource partitioning strategies, demonstrating good adaptability. In addition, compared with the Even strategy, the average STP of the Dynamic Optimizations+CSDSR strategy is improved by 18.96%. Further analysis revealed that for application combinations with good data locality between SMs, such as BP_SPMV, CP_FDTD, SORT_SPMV, and HG_FDTD, the overall throughput improvement of the system is more significant.

[0054] Average Normalized Turnaround Time (ANTT) measures the relative execution latency of each application in a multi-kernel execution environment. A lower ANTT value indicates a smaller relative execution latency for each application, suggesting a more user-friendly overall scheduling strategy. Its calculation is shown in formula (2). Here, n represents the number of kernels currently executing concurrently; the subscript... Indicates the first One kernel, ; Indicates the first The number of execution cycles for each kernel in standalone execution mode; Indicates the first The number of execution cycles for each kernel in a multi-kernel concurrent execution mode. It can be seen that ANTT and STP are negatively correlated to some extent.

[0055] (2) Figure 4The results show the ANTT comparison of different test program combinations under various scheduling strategies. Experimental results indicate that, compared to the Even and Dynamic Optimizations methods, Even+CSDSR (which adds cross-SM data sharing and residency mechanisms) and Dynamic Optimizations+CSDSR reduce the average ANTT by approximately 10.58% and 8.83%, respectively. This demonstrates that CSDSR effectively improves data locality and access efficiency in multi-kernel concurrent scenarios, thereby significantly reducing the waiting time and execution time of each kernel. For example, combinations such as HG_FDTD, CP_FDTD, and BP_SPMV all show ANTT reductions exceeding 20%.

[0056] Figure 5 The power consumption of the four strategies is compared. After adopting CSDSR, the average power consumption of Dynamic Optimizations and the EVEN strategy decreased by 5.94% and 7.4%, respectively.

[0057] In summary, this invention not only effectively improves system throughput and overall system performance, but also has good effects in power consumption optimization, demonstrating its good adaptability in multi-kernel environments while balancing performance and energy consumption control.

[0058] This invention proposes a cross-SM data sharing and residency method, CSDSR, for multi-Kernel scenarios. By analyzing the historical access behavior of the on-chip L1D cache, a cross-SM shared data table is constructed. When an L1D cache block misses, shared data is retrieved from other SMs. Combined with data residency and cold / hot replacement mechanisms, this improves data reuse rate and cache hit rate between SMs. Experimental results show that CSDSR effectively reduces the on-chip cache miss rate, allowing more L1D cache accesses to hit within or between SMs, thus significantly improving on-chip cache access efficiency and ultimately enhancing the overall performance of GPGPUs in multi-Kernel concurrent scenarios. Simultaneously, it also reduces overall system power consumption to some extent.

[0059] Table 1 System Hardware Resource Configuration Table Table 2 Benchmark Test Procedures and Sources The following is an example of the construction and execution of a cross-SM data sharing table (CSDT) based on GPGPU-SIM: Within the GPGPU-SIM simulation framework, the on-chip cache structure is extended to implement the Cross-SM Data Sharing Table (CSDT) proposed in this invention. Its specific components include: KSMT (Kernel-SM Mapping Table): A new data structure added to the GPGPU-SIM on-chip cache to record the SM range allocated to each kernel and the starting position of the SM's entry in the STT (Shared Tag Table). STT (Shared Tag Table): A shared tag table added to the L1D cache module of each SM, containing Tag, Access Count (AC), Valid Bit (V), and Resident Bit Association fields. L1D cache field extension: The FC, BC sampling fields, and lock bit L fields are extended in the GPGPU-SIM's cache_block_t. Cross-SM data path simulation: Logic for querying the CSDT is added to the L1 Dmiss access path, and the cross-SM data return process is simulated.

[0060] Two task kernels (KernelA and KernelB) are simultaneously submitted to GPGPU-SIM for execution. The scheduler allocates different numbers of SMs to the two kernels according to the SM partitioning strategy and initializes KSMT. During the simulation, the FC / BC fields of the L1D cache are counted in each sampling period to identify highly reusable shared data. Based on the correspondence between kernels and SMs in the KSMT table, the highly reusable shared data of the SM is written to STT, and the resident bit L of the cache line containing the data is set to 1. If an L1 Dmiss occurs in the SM, the CSDT is queried first: if it is a hit, the data is returned directly from the shared SM; if it is a miss, the next level cache is accessed. After the kernel execution is completed, the corresponding STT table entry is located through KSMT, its valid bit is cleared and the resident bit is released, and the KernelID that has been executed in KSMT is set to -1.

[0061] In tests conducted with a GTX 2060 configuration on GPGPU-SIM, including multi-kernel parallel scenarios such as matrix multiplication and sparse matrix operations, the L1D cache miss rate decreased by an average of 19%–32%, and the system throughput increased by an average of 18.96%. The overall execution time is reduced by an average of 8%–15%. This implementation result demonstrates that the cross-SM data sharing and residency mechanism of the present invention can effectively reduce high-latency data access caused by L1D cache misses in SMs in a simulated environment, thereby indirectly improving the cache hit rate.

[0062] The verification and replacement strategy effect analysis of the cross-SM data sharing and resident mechanism based on the multi-Kernel environment are as follows: Based on the previous verification, the basic architecture of the L1D cache of GPGPU-SIM is extended to support the resident lock bit L proposed in this invention: the cache line with L=1 cannot be replaced; when the STT table entry is replaced, the L of the corresponding cache line is cleared to achieve resident release; STT adopts the AC minimum table entry replacement algorithm.

[0063] Taking the concurrent execution of tasks by two kernels as an example, multiple kernels execute simultaneously, and the SM corresponding to each kernel accesses a large amount of data, some of which is shared data. The system automatically writes the tags of frequently accessed shared L1D blocks into the STT and locks the corresponding cache lines to keep them resident, so that they will not be replaced in a short period of time. When the STT space is insufficient, the table entries with the minimum AC value are replaced, and the resident lock bits corresponding to the replaced table entries are cleared. As the kernel execution ends, KSMT locates all its corresponding STT table entries, sets the resident bits to 0, and restores the normal replacement state of the L1D cache.

[0064] The comprehensive performance evaluation for multi-kernel hybrid memory access scenarios is as follows: Three typical kernel types are selected in GPGPU-SIM, including: memory throughput intensive (such as BFS, SpMV); and compute intensive (such as SGEMM). Both types of kernels are submitted for execution simultaneously, forming a complex concurrent memory access mode.

[0065] In the simulation environment, the existence of shared data access modes in each kernel is recorded. CSDT automatically identifies shared blocks across kernels based on FC / BC statistics. Residency and release operations are repeatedly triggered under mixed memory access mode to observe the stability of the STT replacement algorithm. System throughput (STP), average normalized turnaround time (AMT), power consumption, and other metrics are recorded.

[0066] Experiments show that in high-concurrency mixed memory access scenarios, this invention still maintains stable benefits: the average system throughput STP is improved by 18.96%. Compared with the Even and Dynamic Optimizations methods, the average ANTT of Even+CSDSR, which adds cross-SM data sharing and residency mechanisms, and Dynamic Optimizations+CSDSR is reduced by approximately 10.58% and 8.83%, respectively. At the same time, its power consumption is reduced by an average of approximately 3.58%, indicating that the CSDSR method of this invention can not only effectively improve system throughput and overall system performance, but also has good effects in power consumption optimization, demonstrating its good adaptability in multi-kernel environments while balancing performance and energy consumption control.

[0067] The proposed method for cross-SM data sharing and residency in multi-kernel scenarios for GPGPU addresses the real-world problem of on-chip cache contention. This method not only accurately identifies highly reusable shared data generated by different kernels during operation but also enables rapid data reuse across SMs through a Cross-SM Data Sharing Table (CSDT), reducing access to the next-level cache and global memory. Simultaneously, the residency mechanism designed in this invention continuously maintains the effective residency of important data in the L1D cache in a multi-kernel concurrent environment, increasing the hit rate of some SMs accessing on-chip cache data and significantly improving access latency.

[0068] This invention boasts strong simulation feasibility: it can fully verify the impact of cross-SM data sharing and residency mechanisms on the execution performance of multiple kernels on a software platform. It can guide hardware optimization: simulation results can provide quantitative references for future GPGPU on-chip cache architecture optimization, reducing hardware implementation risks. It has high scalability: applicable to kernel concurrency scenarios of different scales, the performance of the solution in practical applications can be evaluated by adjusting simulation parameters. It provides a reference for engineering applications: although currently in simulation verification, this method can directly guide chip design, driver optimization, and scheduling strategies, providing a clear and feasible path for subsequent practical deployment.

[0069] Based on the same inventive concept, this invention also provides a GPGPU cross-SM data sharing system in a multi-Kernel scenario, comprising: The L1D cache module is used to add access counters and residency identifiers to the L1D Cache lines of data in each streaming multiprocessor (SM) of the target general-purpose graphics processing unit (GPGPU) in a multi-kernel scenario. The access counters count the access frequency of the L1D Cache lines within a set time period. Based on the access frequency, the frequently accessed L1D Cache lines in each streaming multiprocessor are selected and recorded as high-frequency shared data. Residency operations are performed on these high-frequency shared data according to the residency identifiers, and the unique identifier of the high-frequency shared data and its associated streaming multiprocessor information are recorded.

[0070] The shared tag table module is used to design a cross-SM data sharing table for the target GPGPU. The cross-SM data sharing table includes: a unique identifier for high-frequency shared data, the number of cross-cell accesses of high-frequency shared data, and a valid bit. The valid bit is used to mark the availability status of the corresponding high-frequency shared data.

[0071] The Kernel and SM mapping table module is used to obtain the kernel corresponding to each SM during concurrent GPGPU execution and the storage address of each SM in the cross-SM data sharing table, and construct a global mapping table. When a certain SM has missing computation data, the global mapping table is used to query whether the data is in other SMs. If the target computation data is found and its valid bit is marked as available, the target computation data is directly obtained from the corresponding SM to achieve data sharing.

[0072] For specific limitations on the computing system regarding the GPGPU cross-SM data sharing method in multi-Kernel scenarios, please refer to the limitations mentioned above, which will not be repeated here. Each module in the aforementioned GPGPU cross-SM data sharing system in multi-Kernel scenarios can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0073] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Furthermore, the above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for data sharing across SMs in a multi-Kernel scenario using GPGPU, characterized in that, include: In a multi-kernel scenario, an access counter and a residency identifier are added to the L1D Cache line data of each streaming multiprocessor (SM) in the target general-purpose graphics processor (GPGPU). The access counter counts the access frequency of the L1D Cache line data within a set time period. Based on the access frequency, the L1D Cache line data that is accessed frequently in each streaming multiprocessor is selected and recorded as high-frequency shared data. A residency operation is performed on the high-frequency shared data according to the residency identifier, and the unique identifier of the high-frequency shared data and its corresponding streaming multiprocessor information are recorded. A cross-SM data sharing table is designed for the target GPGPU. The cross-SM data sharing table includes: a unique identifier of high-frequency shared data, the number of cross-cell accesses of high-frequency shared data, and a valid bit. The valid bit is used to mark the availability status of the corresponding high-frequency shared data. When GPGPU executes concurrently, the kernel corresponding to each SM and the storage address of each SM in the cross-SM data sharing table are obtained, and a global mapping table is constructed. When a certain SM has missing computation data, the global mapping table is used to query whether the data is in other SMs. If the target computation data is found and its valid bit is marked as available, the target computation data is directly obtained from the corresponding SM to achieve data sharing.

2. The method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 1, characterized in that, When a certain SM has missing computational data, the method further includes: if the target computational data is not found, it is obtained from the lower-level storage unit through the traditional data access path; when the cross-SM data sharing table reaches the storage limit, the resident identifier is set to the unlocked state and the high-frequency shared data with the fewest cross-unit accesses is deleted; when the kernel execution of the target general-purpose graphics processor ends, the cross-SM data sharing table is cleared.

3. The method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 1, characterized in that, The access counter is configured in the L1D Cache line, specifically including: the access counter is a dual-time access counter, divided into a first-half access counter FC and a second-half access counter BC; FC is used to count the access frequency of L1D Cache line data in the first half of the set sampling period, and BC is used to count the access frequency in the second half of the same sampling period; only when the frequencies counted by FC and BC both reach the preset high-frequency threshold, the corresponding L1D Cache line data is selected as high-frequency shared data.

4. The method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 1, characterized in that, When a certain SM has missing computational data, the global mapping table is used to query whether the data is in other SMs. If the target computational data is found and its valid bit is marked as available, the target computational data is directly obtained from the corresponding SM to achieve data sharing. Specifically, when a certain SM has missing computational data, the SM is recorded as a data request unit. The data request unit locates the SM to which the target missing data belongs through the global mapping table and sends a data access request to the target SM. The target SM verifies the residence status and valid bit of the high-frequency shared data in its own L1D Cache. If the status is valid, the target SM directly reads the data from its own L1D Cache and transmits it to the request unit, while accumulating the cross-unit access count of the data in the cross-SM data sharing table.

5. The method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 1, characterized in that, The valid bit of the cross-SM data sharing table is identified by binary. The specific rules are as follows: when the valid bit is 1, it indicates that the high-frequency shared data is available and the corresponding L1D Cache line is in the resident state; when the valid bit is 0, it indicates that the high-frequency shared data is unavailable and the corresponding L1D Cache line is in the unlocked state; the valid bit status and the resident identifier status are linked in real time and updated synchronously when the status changes.

6. The method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 1, characterized in that, The global mapping table specifically includes: a kernel identifier field, used to record the kernel numbers of each kernel executed concurrently by the general-purpose graphics processor; a streaming multiprocessor (SM) identifier field, used to associate the SM number corresponding to each kernel; and a storage address field for the cross-SM data sharing table, used to record the physical address of the corresponding SM cross-SM data sharing table in on-chip storage. The kernel identifier field allows for quick indexing of the corresponding SM and cross-SM data sharing table, shortening query time.

7. A method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 2, characterized in that, When the cross-SM data sharing table reaches its storage limit, the process of setting the resident identifier to the unlocked state and deleting the most frequently accessed shared data across units includes: traversing the data entries of the current SM in the cross-SM data sharing table, filtering out one or more data entries with the fewest cross-SM data accesses; verifying whether the L1D Cache row data corresponding to the filtered entry is in an unaccessed state; for unaccessed data, first setting the resident identifier to the unlocked state and the valid bit to 0, then deleting the entry from the cross-SM data sharing table, and simultaneously releasing the corresponding storage space of the L1D Cache.

8. A method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 2, characterized in that, When the kernel of the target general-purpose graphics processor finishes execution, the cross-SM data sharing table is cleared. Specifically, this includes: clearing all entries of the current SM in the cross-SM data sharing table, resetting the storage pointer of the cross-SM data sharing table; setting the resident identifiers corresponding to all L1D Cache lines that were previously resident in the SM to the unlocked state; and uniformly updating the valid bits of all associated high-frequency shared data to 0 to ensure that no invalid resident data remains.

9. A method for GPGPU cross-SM data sharing in a multi-Kernel scenario according to claim 1, characterized in that, The L1D Cache row data also includes a row address field, which is used to store the physical address of the L1D Cache corresponding to the frequently shared data. When it is necessary to unlock or delete data, the target L1D Cache row can be directly located through this address field. When accessing data across SMs, the target SM can quickly read the data in the L1D Cache through this address without traversing the cache.

10. A system for implementing the GPGPU cross-SM data sharing method in a multi-Kernel scenario as described in claim 1, characterized in that, include: The L1D cache module is used to add access counters and residency identifiers to the L1D cache lines of data in each streaming multiprocessor (SM) of the target general-purpose graphics processing unit (GPGPU) in a multi-kernel scenario. The access counters count the access frequency of the L1D cache lines within a set time period. Based on the access frequency, the frequently accessed L1D cache lines in each streaming multiprocessor are selected and recorded as high-frequency shared data. Residency operations are performed on these high-frequency shared data according to the residency identifiers, and the unique identifier of the high-frequency shared data and its associated streaming multiprocessor information are recorded. The shared tag table module is used to design a cross-SM data sharing table for the target GPGPU. The cross-SM data sharing table includes: a unique identifier of high-frequency shared data, the number of cross-cell accesses of high-frequency shared data and a valid bit. The valid bit is used to mark the availability status of the corresponding high-frequency shared data. The Kernel and SM mapping table module is used to obtain the kernel corresponding to each SM during concurrent GPGPU execution and the storage address of each SM in the cross-SM data sharing table, and construct a global mapping table. When a certain SM has missing computation data, the global mapping table is used to query whether the data is in other SMs. If the target computation data is found and its valid bit is marked as available, the target computation data is directly obtained from the corresponding SM to achieve data sharing.

Images