System for allocating memory with different properties to shared data objects

A hybrid stack data structure in parallel processors addresses memory management inefficiencies by using multiple memory types, optimizing latency and capacity for shared data objects, thereby enhancing performance and robustness.

DE102013018378B4Active Publication Date: 2024-12-12NVIDIA CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
DE102013018378
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2012-12-21
Filing Date
2013-11-04
Publication Date
2024-12-12
Estimated Expiration
2033-11-04

AI Technical Summary

Technical Problem

Existing parallel processor architectures face inefficiencies in memory management when shared data objects are stored in shared memory, particularly due to the disadvantages of using a single type of memory for stack data structures, which can lead to overflow or increased latency depending on the frequency and size of object access.

Method used

A hybrid stack data structure is implemented using multiple types of physical memories, allowing for efficient allocation and deallocation of shared data objects based on their characteristics, balancing latency and capacity by utilizing block-shared memory for smaller objects and global shared memory for larger ones.

Benefits of technology

This approach enhances memory management by reducing average access time and improving program performance by leveraging the strengths of different memory types, ensuring robustness and flexibility in handling varying data object sizes and access frequencies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A system for allocating shared memory (110) with different properties to shared data objects (306), wherein the system has multiple strand processors (106), with: a hybrid stack generation unit (302) configured to generate in the shared memory (110) a hybrid stack data structure with a lower sub-area having a more favorable property and with an upper sub-area having a less favorable property; and to allocate memory space for a shared data object in the lower sub-area to a data object allocation unit (304) connected and configured with the hybrid stack generation unit (302) if the lower sub-area has sufficient remaining capacity to accommodate the shared data object, and alternatively to allocate memory space for the shared data object in the upper sub-area if the lower sub-area does not have sufficient remaining capacity to accommodate the shared data object (306), wherein the release and allocation of the shared data objects only takes place during a sequential section of a program in which only one strand processor is active, and wherein the shared data objects (306) are only accessed during a parallel section of the program in which multiple strand processors are active.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL AREA

[0001] This application generally concerns parallel processors and, in particular, a system for allocating memory with different properties to shared data objects. BACKGROUND

[0002] As experts in this field know, applications can be run in parallel to improve their performance. Data-parallel applications execute the same process on different data simultaneously. Task-parallel applications execute different processes on the same data simultaneously. Static parallel applications have a degree of parallelism that can be determined before execution. In contrast, the parallelism achievable by dynamic parallel applications can only be determined during execution. Regardless of whether the application is data-parallel, task-parallel, static, or dynamic parallel, it can be executed in a pipeline or parallel processing line, which is common for graphics applications.

[0003] Certain computing systems, such as a single-instruction multi-data (SIMD) processor, are particularly adept at executing data-parallel applications. A pipeline control unit in the SIMD processor creates groups of execution strands and schedules them for execution, during which all strands in the group execute the same instruction simultaneously. In a specific processor, each group has 32 strands, corresponding to 32 execution pipelines or pathways within the SIMD processor.

[0004] Consider a parallel programming model with branching-union or splitting-union, such as OpenMP or OpenACC, implemented in a parallel processing computer system. In this model, certain parts of a program's code are executed by only a single strand (a "parent strand" or "master" strand), while other parts are executed in parallel by multiple strands ("worker" strands). When execution begins, only the parent strand is active. In a work creation construct, execution is split when the parent strand activates worker strands and assigns each worker an "execution task," such as a certain number of iterations of a loop. Worker strands typically execute their assigned tasks in parallel.Once the worker strands have finished, they deactivate, and execution merges when the parent strand resumes execution of the rest of the program code. The duration of program execution when only a single strand is active is referred to herein as the sequential section or sequential phase, and the duration of program execution when more than one strand is active is referred to herein as the parallel section or parallel phase.

[0005] In many partition-union models, including OpenMP and OpenACC, data objects reserved in sequential areas can be accessed in parallel areas. Consequently, parallel processor architectures provide memory for storing data objects that multiple strands can access, if necessary, during execution. This memory can be characterized by many properties, including size, latency, volatility, and others, along with their associated advantages and disadvantages. US 2011 / 0 191 522 A1 describes a flash memory with a cache memory component, and US 6 088 770 A discloses a multiprocessor architecture with shared memory. US 2006 / 0095675 A1 describes a hybrid stack structure with registers and memory, where the registers are loaded first, followed by the memory.

[0006] The object of the invention is to improve memory management in multiprocessor systems when shared data objects are stored in a shared memory. OVERVIEW

[0007] This problem is solved by the features of independent claim 1. The dependent claims describe preferred embodiments. One aspect provides a system for allocating shared memory with different properties to shared data objects and a hybrid stack data structure according to claim 1. In one embodiment, the system includes, among other things: (1) a hybrid stack generation unit configured to generate a hybrid stack data structure in the shared memory, comprising a lower sub-area with a favorable property and an upper sub-area with a less favorable property, and (2) a data object allocation unit connected to the hybrid stack generation unit and configured to allocate memory space for a shared data object in the lower sub-area if the lower sub-area has sufficient remaining capacity to accommodate the shared data object, and alternatively, to allocate memory space for the shared data object in the upper sub-area if the lower sub-area does not have sufficient remaining capacity to accommodate the shared data object. SHORT DESCRIPTION

[0008] Reference is now made to the following descriptions in connection with the accompanying drawings, in which: Fig. 1 a block view of a parallel processor configured to include or execute a system or procedure for allocating memory with different properties to shared data objects; Fig. 2 is a diagram of an embodiment of a hybrid stack; Fig. 3 is a block view of an embodiment of a system for allocating memory of different latency to shared data objects; and Fig. Figure 4 is a flowchart of a procedure for allocating memory of different properties to shared data objects. DETAILED DESCRIPTION

[0009] As previously stated, parallel processor architectures are characterized by many properties. For example, memory can be divided into a larger memory with higher latency and a smaller memory (perhaps by an order of magnitude) with lower latency. Architectures often contain multiple memory blocks with diverse properties and purposes. Multiple strands frequently need to access this memory to share data objects.

[0010] In many partition-union programming models, data objects reserved in the sequential areas can be accessed in the parallel area. However, in certain execution platforms, strand-local memory is not available to other strands. For example, in some commercially available graphics processing units (GPUs), data objects located in the local memory of one strand cannot be accessed by another strand. Thus, a compiler or runtime system that translates a software program into an executable form is responsible for allocating memory to potentially shared objects so that strands active in the parallel area can access them.

[0011] Data objects allocated during the sequential section and local to a function can be conveniently arranged on a batch data structure. However, it is recognized that implementing a batch data structure for shared data objects in only one type of memory (for example, only in globally shared memory, or only in block-shared memory) has disadvantages. If the batch data structure is placed entirely in block-shared memory, which tends to be smaller, the stack can easily overflow, and the program would have to terminate. Conversely, if the batch data structure is placed entirely in globally shared memory, which tends to have a longer processing time, the stack can overflow.If the shared objects have latency, the program's performance would be negatively affected if frequent access to these objects is required, as the average access time would increase. This could be particularly detrimental if a program only has a few small shared objects.

[0012] This paper introduces various embodiments of a system and method for allocating memory with different properties to shared data objects. The embodiments of the system and method generally aim to construct a hybrid stack data structure, i.e., a stack data structure implemented with multiple types of physical memory. The embodiments of the system and method advantageously exploit the strengths of each type of memory, resulting in a mechanism that is less restrictive and achieves better performance than would be possible if only a single type of memory were used. For example, in certain embodiments of the hybrid stack, it is possible to retain the latency advantage of using block-shared memory when relatively few objects are allocated to the stack, which is often the case.However, a hybrid stack allows robustness when many large objects are allocated to the stack by gradually "reverting" to the use of a global shared memory with higher latency.

[0013] Various embodiments of the system and the method can be used in any conventional or yet-to-be-developed parallel programming system in which shared objects are accessed by multiple strands and the work platform provides multiple physical memories with different resource constraints and performance characteristics.

[0014] In some embodiments, the memory segments are configurable, for example, as a compile-time parameter or a runtime parameter. This allows users to decide how best to balance memory resource utilization and program performance. For example, allocating more objects in the memory segment shared by blocks typically reduces the average access time and thus improves program behavior. Similarly, in certain embodiments, the size of each layer of the hybrid stack is configurable, with the bottom layer being the smallest, the next layer larger, and the top layer the largest.

[0015] Several embodiments are presented and described below. Many of the presented and described embodiments initially appear to be specifically for hybrid stacks that have segments arranged in two different storage devices. However, those skilled in the art will recognize that many of these embodiments, together with other embodiments, are intended for hybrid stacks that have more than two segments and are therefore arranged in more than two storage devices. The invention is not limited with respect to the number of different storage devices that can accommodate a hybrid stack.

[0016] Before describing various embodiments of the system and method, a parallel processor is described that is capable of containing or executing a system and method for allocating memory with different properties to shared data objects.

[0017] Fig. Figure 1 is a block view of a parallel processor 100. The parallel processor 100 comprises several strand processors or cores 106, which are divided into strand groups 104 or "bulges" or "chains." The parallel processor 100 contains J strand groups 104-1 to 104-J, each K of which has cores 106-1 to 106-K. In certain embodiments, the strand groups 104-1 to 104-J may be further divided into one or more strand blocks 102. Certain embodiments include thirty-two cores 106 per strand group 104. Other embodiments may contain fewer than four cores in a strand group up to several tens of thousands. Certain embodiments divide the cores 106 into a single strand group 104, while other embodiments have hundreds or even thousands of strand groups 104.Alternative embodiments of the parallel processor 100 may divide the cores 106 only into the strand groups 104, omitting the division at the strand block level.

[0018] The parallel processor 100 further comprises a pipeline control unit 108, a block-shared memory 110, and an array of local memories 112-1 to 112-J, which is assigned to the strand groups 104-1 to 104-J. The pipeline control unit 108 distributes tasks to the various strand groups 104-1 to 104-J via a data bus 114. The cores 106 in a strand group 106-J operate in parallel with each other. The strand groups 104-1 to 104-J communicate with the block-shared memory 110 via a memory bus 116. The strand groups 104-1 to 104-J each communicate with the local memories 112-1 to 112-J via local buses 118-1 to 118-J. For example, a string group 104-J uses the local memory 112-J by means of communication via a local bus 118-J.Certain embodiments of the parallel processor 100 allocate a shared portion of the block-shared memory 110 to each strand block 102 and allow access to the shared portions of the block-shared memory 110 for all strand groups 104 within a strand block 102. Certain embodiments include the strand groups 104 in such a way that they use only the local memory 112. Many other embodiments include the strand groups 104 in such a way that they handle the use of the local memory 112 and the use of the block-shared memory 110 in a balanced manner.

[0019] The embodiment from Fig. Parallel Processor 100 comprises a master strand group, or parent strand group, 104-1. Each of the remaining strand groups, 104-2 through 104-J, is considered a "worker" strand group. The parent strand group 104-1 contains numerous cores, one of which is a parent core, 106-1, that ultimately executes a parent strand. Programs executed in the parallel processor 100 are structured as a sequence of kernels. Typically, each kernel completes its execution before the next kernel begins. In certain embodiments, the parallel processor 100 can execute multiple kernels in parallel, depending on the size of the kernels. Each kernel is structured as a hierarchy of strands to be executed in the cores 106.

[0020] In the following example, and without loss of generality, a GPU commercially available from Nvidia, Santa Clara, California, is used as the execution platform, and OpenACC is used as a partition / union parallel programming model. As previously described, the Nvidia GPU provides global and shared memory, both of which allow access from all strands that are part of the same strand block. The access time for the global shared memory is more than an order of magnitude greater than the access time for the block-shared memory. The size of the global shared memory is more than an order of magnitude greater than the size of the block-shared memory.Table 1 below gives an example of an OpenACC program in which strands in the parallel area access shared variables that are assigned in the sequential area.

[0021] In the example program in Table 1, the parallel constructs with split / union are "pragma acc loop worker" and "pragma acc parallel", and the shared data variables are "arr" and "scalar".

[0022] The `pragma acc parallel` construct activates a single parent strand per "group," and the `pragma acc loop` construct activates other worker strands within a group. Before the other strands in a group are activated, only a single strand of the group is active (the parent strand). The function `foo` is called by the parent strand. The addresses of the local variables of the functions `arr` and `scalar` are passed to the function `bar`, which activates the worker strands. The worker strands then access the `arr` and `scalar` objects.

[0023] For correct execution, the variables "arr" and "scalar" should be allocated to a memory area accessible to the worker strands. In this example, the variables are allocated to a hybrid data stack. The hybrid data stack is implemented using two pre-allocated buffers: one in block-shared memory and one in global shared memory. A variable is allocated to the block-shared memory segment if space is available at the time of allocation to the hybrid stack; otherwise, it is allocated to the global shared memory. Processing information is maintained regarding the maximum sizes of the two segments, the currently active segment, and the position within the active segment where the next stack allocation should be attempted.

[0024] Fig. Figure 2 is a diagram of an embodiment of a hybrid stack 200 with a lower sub-area 202 and an upper sub-area 204. In certain embodiments, the lower sub-area 202 exists in a memory segment shared by blocks with lower latency, and the upper sub-area 204 exists in a globally shared memory segment with higher latency. A stack bottom 206 exists in the lower sub-area 202 and represents the first available or lowest memory address in the hybrid stack 200. Similarly, a stack top 208 exists in the upper sub-area 204 and represents the last available or highest memory address in the hybrid stack 200.

[0025] Allocations or reservations are first made at the bottom of the stack 206 in the lower sub-area 202. As the allocations increase, the lower sub-area fills up until a maximum allocation in the lower sub-area 210 is reached. The maximum allocation in the lower sub-area 210 is essentially the boundary between the lower and upper, or in certain embodiments, between the block-shared and globally shared segments of the hybrid stack. In alternative embodiments, the maximum allocation in the lower sub-area 210 is specified at runtime; and in other embodiments, at compile time.This specification is given by dimensioning the memory segments of the lower sub-area and the memory segment of the upper sub-area in such a way as to achieve a suitable balance between stacking characteristics and performance behavior, such as capacity and stacking latency.

[0026] In certain embodiments where memory of different sizes and latencies is used, it is often the case that the hybrid stack 200 does not exceed the maximum allocation in the lower sub-area 210; some programs do not use sufficiently large amounts of data shared by strands to fill the smaller, lower-latency memory segment of the hybrid stack 200. When the lower sub-area 202 of the hybrid stack 200 is full, the next allocation in the hybrid stack 200 occurs at the lowest address of the upper sub-area 204. The term "lowest" refers to the position in the hybrid stack 200 immediately above the maximum allocation in the lower sub-area 210. As allocations increase, the upper sub-area 204 fills up, and the overall latency of the hybrid stack 200 increases. When memory is freed in the hybrid stack 200, the release first occurs in the upper sub-area 204, i.e.,in the global shared memory segment with the higher latency. As memory continues to be released, the latency decreases, and the global shared memory segment empties, eventually reaching the maximum allocation in the lower sub-area 210. Further releases then affect the lower sub-area 202, i.e., the memory shared by blocks.

[0027] Table 2 below describes an implementation of a hybrid stack with a special set of management variables. The pseudocode for assigning and releasing variables in the hybrid stack is also provided. The attributes "_shared_" and "_device_" indicate that the variable should be assigned according to shared memory or global GPU memory, respectively. Note that this simplified implementation does not address any alignment requirements for the object assigned to the stack.

[0028] In Table 2, “shared_mem_top” and “global_mem_top” denote the position in the shared and globally shared memory segments, respectively, where the allocation should be attempted. “using_shared_mem_segment” is a Boolean indicator that specifies whether the memory segment shared by blocks should be used for the allocation.

[0029] Data object assignment in a hybrid stack "allocate_on_hybrid_stack" allocates memory in the block-shared memory segment if sufficient space is available in the block-shared memory segment; otherwise, the object is allocated in the globally shared memory segment. “deallocate_from_hybrid_stack” sets the stack pointer in the appropriate memory segment and can fall back to using the block-shared memory segment if the globally shared memory segment is completely empty (i.e., no objects are currently allocated in that segment).

[0030] The allocation and release of memory shared by blocks in the hybrid stack only occurs during the sequential part of the program, so there are no strand safety issues during allocation and release.

[0031] Fig. Figure 3 is a block view of an embodiment of a system 300 for allocating memory with different latencies to shared data objects 306. The system 300 comprises a hybrid stack generation unit 302 and a data object allocation unit 304. The hybrid stack generation unit 302 is configured to generate the hybrid stack 200 from Fig. to generate 2. The hybrid stack 200 contains, as in Fig. Figure 3 shows a memory segment 308 shared by blocks and a globally shared memory segment 310, i.e., the lower and upper sub-areas. Fig. 2.

[0032] The data object allocation unit 304 operates on the shared data objects 306 to store the shared data objects 306 in the hybrid stack 200. In some embodiments, the data object allocation unit 304 operates at compile time, while in other embodiments, the data object allocation unit 304 operates at runtime. The data object allocation unit 304 favors the allocation of the block-shared memory segment 308. At allocation time, the data object allocation unit 304 checks whether the block-shared memory segment 308 has available capacity to store a data object. If capacity is available in the block-shared memory segment 308, the data object is stored in the block-shared memory segment 308.If there is no capacity in the block-shared memory segment 308, the data object is stored in the globally shared memory segment 310.

[0033] Fig.Figure 4 is a flowchart of a procedure for allocating memory with different properties to shared data objects. The procedure begins in a start step 410. In step 420, a hybrid stack data structure is created in the shared memory, which has a lower sub-area with a more favorable property and an upper sub-area with a less favorable property. At this point, allocations are made in the data structure for shared data objects, which must contain an arbitrary strand. Consequently, in a decision step 430, it is determined whether the lower sub-area has sufficient remaining capacity to accommodate a given required shared data object. If so, memory for the shared data object is allocated in the lower sub-area in step 440.If this is not the case, storage space for the shared data object is alternatively allocated in the upper sub-area in step 450. In step 460, storage space is released for any shared data objects that no longer need to be included in a strand of the data structure. The procedure ends in a final step 470.

Claims

[1] A system for allocating shared memory (110) with different properties to shared data objects (306), the system comprising a plurality of thread processors (106), comprising: a hybrid stack generation unit (302) configured to generate in the shared memory (110) a hybrid stack data structure having a lower sub-area with a more favorable property and an upper sub-area with a less favorable property; and a data object allocation unit (304) connected to the hybrid stack generation unit (302) and configured to allocate storage space for a shared data object in the lower sub-area if the lower sub-area has sufficient remaining capacity to accommodate the shared data object, and alternatively to allocate storage space for the shared data object in the upper sub-area if the lower sub-area has insufficient remaining capacity to accommodate the shared data object (306), wherein the shared data objects are released and allocated only during a sequential portion of a program in which only one thread processor is active, and the shared data objects (306) are accessed only during a parallel portion of the program in which multiple thread processors are active. [2] The system of claim 1, wherein the more favorable characteristic is a smaller latency and the less favorable characteristic is a larger latency. [3] The system of claim 2, wherein the lower latency portion is a block shared memory (308), the upper latency portion is a global shared memory (310), and the system is operable in a parallel local memory computing system. [4] The system according to any one of claims 1-3, wherein the lower portion is at most one order of magnitude smaller than the upper portion. [5] The system according to any one of claims 1-4, wherein the size of the lower portion and the upper portion is configurable by software. [6] The system according to any one of claims 1-5, wherein the hybrid stack generation unit (302) and the data object allocation unit (304) are implemented in a component selected from the group comprising: a compiler, and a runtime system. [7] The system of any of claims 1-6, wherein the data object allocation unit (304) is operable only during a sequential portion of program execution.

Citation Information

Patent Citations

  • Three stage hybrid stack model

    US20060095675A1

  • Managing Metadata and Page Replacement in a Persistent Cache in Flash Memory

    US20110191522A1

  • Shared memory multiprocessor performing cache coherency

    US6088770A