Method for optimizing memory of streaming data heterogeneous computing based on dynamic scalable memory pool

Abstract

The application discloses a memory optimization method for stream data heterogeneous computing based on a dynamic elastic memory pool, which comprises four modules, namely a CPU end host memory pool module, a GPU end device memory pool module, a memory pool resource unified monitoring module and a stream data memory demand prediction module.The main functions of the four modules are as follows: the host memory pool module applies for locking memory in advance for caching, the device memory pool applies for device memory in advance for caching, and the memory allocation demand of stream data heterogeneous computing is met; the memory pool resource unified monitoring module regularly pulls the memory resource usage of the host memory pool and the device memory pool, the stream data memory demand prediction module predicts the memory allocation demand in a future period of time by monitoring the memory allocation demand in each period of time, and guides the dynamic expansion and contraction of the two memory pools according to the memory pool resource water level condition obtained by the memory pool resource unified monitoring module, so as to cope with the future memory allocation demand of stream data heterogeneous computing.

Description

Technical Field

[0001] The present invention relates to the field of heterogeneous computing with both CPU and GPU, and specifically to a memory optimization method for heterogeneous computing of streaming data based on a dynamically scalable memory pool. Background Technology

[0002] With the development of general-purpose GPUs and CPU-GPU heterogeneous computing, the physical capacity of host memory on the CPU side and device memory on the GPU side of a single heterogeneous computing node is constantly increasing. The memory management strategies of the CUDA unified computing device architecture, designed for general-purpose scenarios, also need to consider the synchronization issues of massively parallel GPUs. The memory allocation performance provided cannot meet the memory allocation performance requirements of large-scale streaming data in heterogeneous computing platforms. Therefore, memory optimization for heterogeneous computing is of great significance for improving the throughput of heterogeneous computing platforms while reducing memory allocation latency.

[0003] Compared to memory allocators, memory pooling technology offers greater flexibility, customizability, and the ability to achieve better performance for specific applications. In large-scale traffic data processing scenarios based on heterogeneous computing, memory pooling technology can implement optimal memory management strategies tailored to the memory allocation characteristics of the algorithms implemented in the current scenario. Furthermore, by pre-allocating memory and delaying memory release, memory pooling technology reduces application memory allocation latency and system memory management performance overhead, thus improving the memory management performance of heterogeneous computing platforms and the throughput of large-scale streaming data processing.

[0004] Existing memory pool technologies employ only passive and static scaling strategies, achieving passive scaling by pre-setting static high-water and low-water parameters. However, the inflow of large-scale streaming data varies over time, with potential sudden increases and decreases in traffic. Traditional passive scaling strategies cannot predict future traffic changes, potentially leading to increased memory allocation latency due to untimely memory pool expansion and low system memory utilization due to untimely memory pool shrinkage. Summary of the Invention

[0005] The purpose of this invention is to provide a memory optimization method for heterogeneous computing of streaming data based on a dynamically scalable memory pool. The method includes:

[0006] (a) Create globally unique CPU-side host memory pools and GPU-side device memory pools based on user-defined configuration files. Configuration items are necessary for passive scaling of the memory pools, used to supplement scenarios where dynamic scaling is not required.

[0007] The specific process of (a) is as follows:

[0008] (a-1) Based on the initial size Count of the memory pool and the initial supported memory allocation size, calculate the total memory size required for initialization by multiplication, and then divide the total memory size by the size of a superblock to obtain the number of superblocks n required for initialization;

[0009] (a-2) The CPU-side host memory pool calls the cudaMallocHost interface n times to request n host memory superblocks. The superblocks are divided into multiple host memory blocks with the size as the initial supported memory allocation size parameter and inserted into the management list of the memory pool for caching.

[0010] (a-3) The GPU-side device memory pool calls the cudaMalloc interface n times to request n device memory superblocks, divides them into multiple device memory blocks with the initial supported memory allocation size parameter as the size, and inserts them into the memory pool's management list for caching;

[0011] (b) Run a UDP-based streaming data receiving server to receive streaming data by listening to UDP sockets.

[0012] (c) Before each stream data reception, the server requests a block of host-locked memory from the CPU host memory pool to serve as a buffer for storing the stream data to be received.

[0013] The specific process of (c) is as follows:

[0014] (c-1) Based on the thread identifier and memory allocation size, map the request to a sub-memory pool one-to-one. If the sub-memory pool does not exist, create the sub-memory pool according to the initial size parameter of the memory pool.

[0015] (c-2) The CPU host memory pool attempts to acquire the mutex lock of the sub-memory pool and checks if there is a free memory block in the memory management list. If not, the passive expansion of the sub-memory pool is performed. The size of the passive expansion is calculated based on the expansion factor of the memory pool.

[0016] (c-3) After determining that there is a free memory block, take a memory block from the memory management list of the sub-memory pool, allocate it to the application, and finally release the mutex lock;

[0017] (d) After storing the streaming data in the host locked memory, the server requests a block of device memory from the GPU device memory pool.

[0018] The specific process of (d) is as follows:

[0019] (d-1) Based on the CUDA stream identifier and memory allocation size, a request is mapped to a sub-memory pool one-to-one. If the sub-memory pool does not exist, it is created based on the initial size parameter of the memory pool.

[0020] (d-2) The GPU-side device memory pool attempts to acquire the mutex lock of the sub-memory pool, checks if there are any free memory blocks in the memory management list, and if not, performs passive expansion of the sub-memory pool. The size of the passive expansion is calculated based on the expansion factor of the memory pool to determine the actual expansion size.

[0021] (d-3) After determining that there is a free memory block, take a memory block from the memory management list of the sub-memory pool, allocate it to the application, and finally release the mutex lock;

[0022] (e) After successful device memory allocation, the streaming data is copied from the host memory to the device memory via the PCIe bus, and then the heterogeneous computing task is submitted to the CUDA system.

[0023] (f) The UDP-based streaming data receiving server continuously executes steps (c)-(e) to achieve heterogeneous computing processing for streaming computation. During this period, the system resource usage and resource level of the memory pool are periodically collected using a "pull" method. Then, based on the ARIMA-based time-series prediction method, the memory allocation demand for the next period is predicted according to the system memory allocation demand over a historical period, guiding the two memory pools to actively scale, or not scale at all.

[0024] The specific process of (f) is as follows:

[0025] (f-1) By monitoring the memory allocation demand in each time period, the memory allocation demand for a future period can be predicted. Specifically, the ARIMA algorithm is used to calculate the memory allocation demand at the next time point based on the three parameters p, d, and q set by the system. The expression for ARIMA(p,d,q) is: y_t=\mu+\\varphi_1y_{t-1}+\cdots+\\varphi_py_{tp}+\\theta_1e_{t-1}+\\cdots+\\theta_qe_{tq}\;

[0026] (f-2) Periodically retrieve the memory resource usage of the host memory pool and device memory pool; then, based on the memory pool resource level and ARIMA prediction results, guide the dynamic expansion and contraction of the two memory pools, or keep them unchanged.

[0027] This invention proposes a memory management system for heterogeneous streaming data computing based on a dynamically scalable memory pool. The system includes:

[0028] The CPU-side host memory pool module is suitable for quickly allocating and releasing CPU-side host locked memory for heterogeneous computing applications of streaming data.

[0029] The GPU-side device memory pool module is suitable for quickly allocating and releasing GPU-side device memory for heterogeneous computing applications of streaming data;

[0030] The unified monitoring module for memory pool resources is suitable for collecting the usage of CPU host memory resources and GPU device memory resources managed by two memory pools in a background thread mode without affecting the performance of the memory pools, and monitoring the resource level of the two memory pools.

[0031] The streaming data memory demand prediction module is suitable for predicting the system memory allocation demand for a future period based on the historical system memory allocation demand using the ARIMA time series prediction method, and for guiding the dynamic expansion and contraction of the two memory pools based on the memory pool resource level status obtained by the unified memory pool resource monitoring module.

[0032] The beneficial effects of this invention are that it provides a memory optimization method for heterogeneous streaming data computing based on a dynamically scalable memory pool. The memory pool includes a CPU-side host memory pool and a GPU-side device memory pool. Based on the principles of advance allocation and delayed release, it significantly reduces the number of CUDA memory allocation API calls by allocating a large superblock at once and then dividing it into smaller memory blocks. Furthermore, by managing memory blocks according to their size as a type, it reduces the time spent searching for available memory blocks during allocation, thereby lowering the memory allocation latency required for heterogeneous streaming data computing. This invention also addresses the problem of untimely memory pool scaling, which is common in traditional passive memory scaling. By introducing a unified memory pool resource monitoring module and a streaming data memory demand prediction module, and using the ARIMA algorithm to predict system memory allocation needs over a future period, it ultimately achieves proactive memory pool scaling in heterogeneous streaming data computing scenarios. This optimizes the problems of increased memory allocation latency caused by untimely memory pool expansion and poor system memory utilization caused by untimely memory pool shrinkage. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the memory pool of the present invention.

[0034] Figure 2 This is a flowchart of the memory pool expansion process of the present invention.

[0035] Figure 3 This is a flowchart of the memory pool operation method of the present invention.

[0036] Figure 4 This is a diagram illustrating the principle of active scaling of a memory pool. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific implementation steps, but this is not intended to limit the invention.

[0038] This invention provides a memory optimization method for heterogeneous streaming data computing based on dynamically scalable memory pools. The method consists of four modules: a CPU-side host memory pool module, a GPU-side device memory pool module, a unified memory pool resource monitoring module, and a streaming data memory demand prediction module. The main functions of these four modules are: the host memory pool module pre-allocates and locks memory for caching, and the device memory pool pre-allocates device memory for caching, satisfying the memory allocation requirements of heterogeneous streaming data computing; the unified memory pool resource monitoring module periodically retrieves the memory resource usage of the host and device memory pools; the streaming data memory demand prediction module monitors the memory allocation demand for each time period to predict the memory allocation demand for the future, and guides the dynamic expansion and contraction of the two memory pools based on the memory pool resource level obtained from the unified memory pool resource monitoring module to cope with future memory allocation demands of heterogeneous streaming data computing.

[0039] Specifically, the implementation method of the above memory optimization method for heterogeneous computing of streaming data based on dynamically scalable memory pools is as follows:

[0040] S1: Creates globally unique CPU-side host memory pools and GPU-side device memory pools based on user-defined configuration files. Configuration items in the configuration file include the initial size of the memory pool, the expansion factor, the shrink factor, the maximum limit for a single expansion, the maximum limit for a single shrink, the high-water mark threshold, and the low-water mark threshold. These configuration items are necessary for passive scaling of the memory pool, used to supplement scenarios where dynamic scaling is unnecessary.

[0041] Specifically, step S1 includes the following sub-steps:

[0042] S1-1: Calculate the total memory size required for initialization by multiplication based on the initial size Count of the memory pool and the initial supported memory allocation size;

[0043] S1-2: Divide the total memory size by the size of a superblock to obtain the number of superblocks n required for initialization;

[0044] S1-3: The CPU-side host memory pool calls the cudaMallocHost interface n times to request n host memory superblocks, divides them into multiple host memory blocks with the initial supported memory allocation size parameter as the size, and inserts them into the memory pool's management list for caching;

[0045] S1-4: The GPU-side device memory pool calls the cudaMalloc interface n times to request n device memory superblocks, divides them into multiple device memory blocks with the initial supported memory allocation size parameter as the size, and inserts them into the memory pool's management list for caching;

[0046] S2: Runs a UDP-based streaming data receiving server, which receives streaming data by listening to UDP sockets.

[0047] S3: Before each stream of data is received, the server requests a block of host-locked memory from the CPU-side host memory pool to serve as a buffer for storing the received stream of data. To quickly allocate the host-locked memory needed by the server, the CPU-side host memory pool performs the following sub-steps:

[0048] S3-1: Based on the thread identifier and memory allocation size, a request is mapped to a sub-memory pool on a one-to-one basis. If the sub-memory pool does not exist, it is created based on the initial size parameter of the memory pool.

[0049] S3-2: The CPU host memory pool attempts to acquire the mutex lock of the sub-memory pool and checks if there is a free memory block in the memory management list. If not, it performs passive expansion of the sub-memory pool. The size of the passive expansion is calculated based on the expansion factor of the memory pool to determine the actual expansion size.

[0050] S3-3: After determining that there is a free memory block, take a memory block from the memory management list of the sub-memory pool, allocate it to the application, and finally release the mutex lock;

[0051] S4: After storing the streaming data in the host's locked memory, the server requests a block of device memory from the GPU's device memory pool. To quickly allocate the device memory needed by the server, the GPU's device memory pool performs the following sub-steps:

[0052] S4-1: Based on the CUDA stream identifier and memory allocation size, a request is mapped to a sub-memory pool one-to-one. If the sub-memory pool does not exist, it is created based on the initial size parameter of the memory pool.

[0053] S4-2: The GPU-side device memory pool attempts to acquire the mutex lock of the sub-memory pool, checks if there are any free memory blocks in the memory management list, and if not, performs passive expansion of the sub-memory pool. The size of the passive expansion is calculated based on the expansion factor of the memory pool to determine the actual expansion size.

[0054] S4-3: After determining that there is a free memory block, take a memory block from the memory management list of the sub-memory pool, allocate it to the application, and finally release the mutex lock;

[0055] S5: After successful device memory allocation, the streaming data is copied from the host memory to the device memory via the PCIe bus, and then the heterogeneous computing task is submitted to the CUDA system.

[0056] S6: The UDP-based streaming data receiving server continuously executes steps S3-S5 to achieve heterogeneous computing processing for streaming computation. During this process, this invention further optimizes the memory system of the heterogeneous computing platform by introducing two additional modules: a unified memory pool resource monitoring module and a streaming data memory requirement prediction module.

[0057] S6-1: The memory resource module periodically retrieves the memory resource usage information from the host memory pool and the device memory pool;

[0058] S6-2: The streaming data memory demand prediction module monitors memory allocation demand for each time period and predicts memory allocation demand for a future period. Specifically, it uses the ARIMA algorithm to calculate the memory allocation demand for the next time point based on three parameters p, d, and q set by the system. The expression for ARIMA(p,d,q) is:

[0059]

[0060] S6-3: The streaming data memory demand prediction module guides the dynamic expansion and contraction of the two memory pools based on the memory pool resource level obtained from the unified monitoring module, in order to meet the future memory allocation needs of heterogeneous streaming data computing.

Claims

1. A memory optimization method for heterogeneous computing of streaming data based on a dynamically scalable memory pool, wherein, The memory pool includes a CPU-side host memory pool and a GPU-side device memory pool. Based on the principles of advance allocation and delayed release, it reduces the number of CUDA memory allocation API calls by allocating a large superblock at once and then splitting it into smaller memory blocks. Furthermore, it reduces the time spent searching for available memory blocks during memory allocation by managing memory blocks according to their size as a type. This reduces the memory allocation latency required for heterogeneous streaming data computing. By introducing a unified memory pool resource monitoring module and a streaming data memory demand prediction module, it predicts the system memory allocation demand for a period of time in the future based on the ARIMA algorithm. This enables proactive scaling of the memory pool in heterogeneous streaming data computing scenarios, optimizing the problem of increased memory allocation latency caused by untimely memory pool expansion and poor system memory utilization caused by untimely memory pool shrinkage. It includes the following steps: S1: Create globally unique CPU-side host memory pool and GPU-side device memory pool based on user-defined configuration files. Configuration items in the configuration file include the initial size of the memory pool, the expansion factor, the shrinkage factor, the maximum limit for a single expansion, the maximum limit for a single shrink, the high-water mark threshold, and the low-water mark threshold. These configuration items are for passive scaling of the memory pool, used to supplement the needs of dynamically scaling scenarios. Specifically, step S1 includes the following sub-steps: S1-1: Calculate the total memory size required for initialization by multiplication based on the initial size Count of the memory pool and the initial supported memory allocation size; S1-2: Divide the total memory size by the size of a superblock to obtain the number of superblocks n required for initialization; S1-3: The CPU-side host memory pool calls the cudaMallocHost interface n times to request n host memory superblocks, divides them into multiple host memory blocks with the initial supported memory allocation size parameter as the size, and inserts them into the memory pool's management list for caching; S1-4: The GPU-side device memory pool calls the cudaMalloc interface n times to request n device memory superblocks, divides them into multiple device memory blocks with the initial supported memory allocation size parameter as the size, and inserts them into the memory pool's management list for caching; S2: Runs a UDP-based streaming data receiving server, which receives streaming data by listening to UDP sockets; S3: Before each stream data reception, the server requests a block of host-locked memory from the CPU-side host memory pool as a buffer to store the received stream data. To quickly allocate the host-locked memory needed by the server, the CPU-side host memory pool performs the following sub-steps: S3-1: Based on the thread identifier and memory allocation size, a request is mapped to a sub-memory pool on a one-to-one basis. If the sub-memory pool does not exist, it is created based on the initial size parameter of the memory pool. S3-2: The CPU host memory pool attempts to acquire the mutex lock of the sub-memory pool and checks if there is a free memory block in the memory management list. If not, it performs passive expansion of the sub-memory pool. The size of the passive expansion is calculated based on the expansion factor of the memory pool to determine the actual expansion size. S3-3: After determining that there is a free memory block, take a memory block from the memory management list of the sub-memory pool, allocate it to the application, and finally release the mutex lock; S4: After storing the streaming data in the host locked memory, the server requests a block of device memory from the GPU device memory pool. To quickly allocate the device memory needed by the server, the GPU device memory pool performs the following sub-steps: S4-1: Based on the CUDA stream identifier and memory allocation size, a request is mapped to a sub-memory pool one-to-one. If the sub-memory pool does not exist, it is created based on the initial size parameter of the memory pool. S4-2: The GPU-side device memory pool attempts to acquire the mutex lock of the sub-memory pool, checks if there are any free memory blocks in the memory management list, and if not, performs passive expansion of the sub-memory pool. The size of the passive expansion is calculated based on the expansion factor of the memory pool to determine the actual expansion size. S4-3: After determining that there is a free memory block, take a memory block from the memory management list of the sub-memory pool, allocate it to the application, and finally release the mutex lock; S5: After successful device memory allocation, the streaming data is copied from the host memory to the device memory via the PCIe bus, and then the heterogeneous computing task is submitted to the CUDA system. S6: The UDP-based streaming data receiving server will continuously execute steps S3-S5 to achieve heterogeneous computing processing for streaming computing. During this process, by introducing two additional modules, namely the unified monitoring module for memory pool resources and the streaming data memory requirement prediction module, the memory system of the heterogeneous computing platform is further optimized. S6-1: The memory resource module periodically retrieves the memory resource usage information from the host memory pool and the device memory pool; S6-2: The streaming data memory demand prediction module monitors the memory allocation demand for each time period and predicts the memory allocation demand for the future. Specifically, it uses the ARIMA algorithm to calculate the memory allocation demand for the next time point based on three parameters p, d, and q set by the system. The expression for ARIMA(p,d,q) is: S6-3: The streaming data memory demand prediction module guides the dynamic expansion and contraction of the two memory pools based on the memory pool resource level obtained from the unified monitoring module, in order to meet the future memory allocation needs of heterogeneous streaming data computing.

Images