Unified memory system based on heterogeneous computing

Abstract

The invention provides a unified memory system based on heterogeneous computing, belongs to the field of parallel computing and artificial intelligence, and realizes memory sharing by constructing a global virtual address space and eliminating explicit data transmission between a host end and an equipment end. The core module comprises: 1) a cache consistency manager, which adopts an adaptive protocol (write broadcast / write invalidation) and a cache directory table (CCDT) to ensure data consistency; 2) an MMU memory management unit which supports dynamic address translation, on-demand distribution and page migration; and 3) an intelligent data migration unit which dynamically migrates the hotspot data to a near-end memory based on the prediction model and reduces the access delay. The architecture is suitable for edge AI applications with high requirements on real-time performance and energy efficiency.

Description

Technical Field

[0001] The present invention relates to the fields of parallel computing and artificial intelligence, and in particular to a unified memory system based on heterogeneous computing, which is suitable for edge AI devices with high real-time requirements and limited resources. Background Art

[0002] In traditional heterogeneous computing architectures, the host and device use independent memory spaces and require explicit data transfer to achieve collaboration. This approach has several drawbacks. First, data needs to be copied multiple times between the host and device, resulting in high data transmission latency. Second, developers need to manually allocate memory and synchronize data, which complicates memory management and is developer-unfriendly. Furthermore, static memory allocation cannot accommodate dynamic workloads, resulting in low resource utilization.

[0003] While existing heterogeneous programming frameworks like CUDA can partially address memory sharing, they are not optimized for edge AI scenarios and cannot meet the low-power and real-time requirements. Furthermore, traditional data migration strategies lack the dynamic awareness of edge scenarios, and cache coherence protocols are not optimized for easy integration between host and device, resulting in high data migration overhead. Summary of the Invention

[0004] To address the above technical issues, the present invention provides a unified memory system based on heterogeneous computing, aiming to improve the real-time performance and energy efficiency of edge AI systems. The objectives are as follows:

[0005] (1) Unified memory addressing: Create a global virtual address space shared by the host and device, implement dynamic address mapping through the hardware-assisted memory management unit (MMU), and eliminate explicit data transfer.

[0006] (2) Intelligent data migration strategy: Based on real-time access pattern monitoring and prediction models, data is dynamically migrated to the optimal storage location, and non-blocking migration is achieved by combining the DMA engine.

[0007] (3) Adaptive cache consistency protocol: Dynamically select write-broadcast or write-invalidate strategies based on the real-time requirements of the task, and introduce a cache directory table (CCDT) to optimize synchronization efficiency.

[0008] The technical solution of the present invention is:

[0009] (1) System architecture: supports a global virtual address space shared by the host and device, including hardware MMU and cache coherence controller.

[0010] (2) Dynamic memory management unit: a data migration module that integrates the prediction model and the demand-driven allocation algorithm.

[0011] (3) Cache consistency protocol: Adaptively select write-broadcast / write-invalidate strategies and combine them with the cache directory table to maintain data consistency.

[0012] (4) Resource sharing method: real-time monitoring of memory access patterns, dynamic adjustment of data location, memory allocation and task scheduling.

[0013] A unified memory architecture system based on heterogeneous computing, comprising:

[0014] a) Global virtual address space module, which creates a shared virtual memory space for the host and device, and supports dynamic address mapping.

[0015] b) Cache consistency manager, including adaptive protocols (write broadcast / write invalidate) and cache directory table (CCDT), to maintain data consistency.

[0016] c) MMU memory management unit, which implements dynamic memory allocation, page migration and prefetch optimization.

[0017] d) Intelligent data migration unit, which performs non-blocking migration through the DMA engine based on access pattern prediction and benefit evaluation.

[0018] Furthermore, the adaptive protocol of the cache consistency manager includes:

[0019] a) Real-time tasks use a write-broadcast protocol to synchronously update all caches through the on-chip network.

[0020] b) Non-real-time tasks use a write-invalidate protocol and only mark other caches as invalid.

[0021] The cache directory table of the cache consistency manager includes:

[0022] Record the cache status and owner information of the data block, and design a state transition machine for different states in hardware; the state transition mechanism implements data consistency management by operating the status and owner fields in CCDT.

[0023] Further,

[0024] The MMU memory management unit supports:

[0025] a) Fine-grained memory allocation (4KB-64KB page blocks) and dynamic recycling.

[0026] b) Memory pressure detection and prefetch optimization based on hardware performance counters (PMCs).

[0027] The MMU memory management unit includes:

[0028] The address translation unit is responsible for converting virtual addresses to physical addresses; it converts 64-bit virtual addresses into physical addresses through the page table structure and allocates an independent virtual address space to each process.

[0029] The memory allocator allocates memory on demand when the task actually needs it, and updates the synchronous address mapping changes in real time, supporting fine-grained allocation.

[0030] Further,

[0031] The intelligent data migration unit specifically includes

[0032] By dynamically tracking memory access patterns, frequently accessed data is migrated to the most appropriate processing unit memory based on workload characteristics, reducing memory access latency.

[0033] Use predictive models to migrate data in advance and avoid unnecessary memory pauses.

[0034] Combined with cache-aware data placement and scheduling, it reduces cache evictions, improves cache hit rates, optimizes bandwidth consumption, and ultimately improves system processing efficiency.

[0035] Specifically, it includes the following modules:

[0036] (1) Data monitoring module: responsible for real-time monitoring of memory access on the host and device sides, and collecting information on data access frequency, access sequence, and spatial locality through hardware performance counters and software probes.

[0037] (2) Analysis and prediction module: Analyze the data collected by the monitoring module and establish a prediction model.

[0038] (3) Decision-making module: Based on the results of the analysis and prediction module, it determines whether data migration is necessary and determines the target location of the migration.

[0039] (4) Migration execution module: This module uses the DMA engine to migrate data from the source location to the target location. This module needs to cooperate with the system's memory management unit (MMU) and cache system to ensure the correctness and efficiency of data migration.

[0040] (5) Information update module: After the data migration is completed, the memory mapping table and cache directory in the system are updated to ensure that the system can correctly access the migrated data.

[0041] Further,

[0042] Data monitoring module: collects CPU / GPU memory access data in real time to provide a basis for intelligent decision-making.

[0043] (1) Memory access pattern: Track the spatial locality, temporal locality (interval between repeated data accesses), and operation type (read / write ratio) of data accesses to identify frequently accessed data blocks.

[0044] (2) Processing unit status: monitors the core utilization, cache hit rate, and task characteristics of the CPU / GPU.

[0045] (3) System resource status: statistics of memory usage and bandwidth usage.

[0046] Analysis and prediction module:

[0047] Data preprocessing: Calculate the "GPU high-frequency data index" = access frequency × (1-cache hit rate)

[0048] Threshold > 100 is considered GPU high-frequency data

[0049] Clustering and grouping: Use K-means to classify memory blocks into three categories: GPU core data, CPU control data, and shared data, and identify data locality characteristics.

[0050] Model prediction and decision-making process:

[0051] Prediction, uses the random forest algorithm to perform a classification prediction model to decide the optimal storage location of the data block, that is, to determine whether the data block should be stored in the CPU memory, GPU memory, or shared cache.

[0052] (1) Input feature information is access frequency, data block size, cache hit rate, task type

[0053] (2) Model training

[0054] (3) Output the “best storage location label” of the data block, and trigger the migration decision when the confidence level is ≥90%.

[0055] Decision: Based on the random forest output, the decision rule is:

[0056] If Random Forest Output == 1 (GPU memory):

[0057] If the remaining space of GPU memory is greater than the data block size:

[0058] Trigger DMA engine migration (priority = 1)

[0059] else:

[0060] Perform memory recovery first (release invalid data blocks), then migrate

[0061] elif output == 0 (CPU memory):

[0062] Prioritize placement in the CPU L3 cache (skip migration if cache hit)

[0063] else(shared cache):

[0064] Place it in the CPU-GPU shared memory area and enable cache consistency protocol

[0065] Through the prediction of the random forest model, the decision module implements multi-level optimization from data placement, prefetching strategy to memory allocation.

[0066] The migration execution module collaborates with the system's memory management unit and cache system to ensure the correctness and efficiency of data migration.

[0067] The decision-making mechanism of the intelligent data migration unit includes:

[0068] a) Migration probability calculation:

[0069] b) Migration benefit assessment;

[0070] c) When there are multiple migration data blocks, migration priority sorting is required.

[0071] Going further,

[0072] The workflow includes the following steps:

[0073] a) Build a global virtual address space to achieve memory sharing between the host and device;

[0074] b) Maintaining data consistency through the cache consistency manager (CCDT);

[0075] c) dynamically optimize data location using intelligent data migration units;

[0076] d) On-demand memory allocation and address translation based on MMU.

[0077] The beneficial effects of the present invention are

[0078] The proposed solution can be applied to edge AI devices such as IoT terminals and in-vehicle computing units, and is applicable to various devices within heterogeneous computing frameworks. Through dynamic optimization technology, it improves device memory utilization, reduces the developer's manual memory management workload, and increases device throughput. This solution has high application value in areas such as autonomous driving and smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0080] Figure 2 It is a schematic diagram of cache consistency module processing logic;

[0081] Figure 3 It is a schematic diagram of the hardware state machine transition;

[0082] Figure 4 It is a logical diagram of intelligent data migration unit processing. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0084] To address the high latency and low efficiency issues of host-device collaborative processing in edge AI systems, the present invention proposes a unified memory architecture (UMA) that collaborates with both software and hardware. Through unified addressing, intelligent data migration, and a hierarchical cache consistency mechanism, it eliminates explicit data transmission and improves the real-time performance and energy efficiency of edge AI systems.

[0085] The system architecture diagram of the unified memory architecture (UMA) for host-device resource sharing in edge AI systems is shown in the attached figure. Figure 1 As shown in the figure, the edge AI device acts as the input layer to collect data, which is then fed into the UMA unified memory architecture on the host and device sides. After a series of data processing, decisions are made and actions are taken based on the data results of the collaborative processing between the host and device sides. The system's host-device unified memory architecture and resource sharing method include the following key modules and steps:

[0086] 1. Build a global virtual address space

[0087] A jointly accessible system-level virtual memory space is created for the host and device sides, allowing both to directly access shared memory under the same memory framework without explicit data copy operations, thereby improving data access efficiency.

[0088] 2. Cache Coherence Manager

[0089] To ensure cache consistency among processor units in a heterogeneous memory architecture, a cache consistency manager is set up to ensure that data modifications made by one processing unit are reflected in the caches of other processing units in a timely manner by setting specific protocols, thus preventing the use of outdated data.

[0090] 3.MMU memory management unit

[0091] It is responsible for dynamic memory allocation and data transfer optimization. It converts virtual addresses into physical addresses through memory mapping, enabling efficient memory management. It also performs operations such as memory prefetching, on-demand allocation, and page migration to optimize memory usage.

[0092] 4. Intelligent data migration unit

[0093] Dynamically migrate data to the most appropriate processing unit memory based on memory access patterns and workload characteristics. Using predictive models, data frequently accessed by a processing unit is moved to its nearby memory area, reducing memory access latency and improving overall system performance.

[0094] In the above, the host side is responsible for data preprocessing, and the device side is responsible for complex parallel computing.

[0095] The second module cache consistency management is described in detail as follows:

[0096] a) Adopting the Adaptive Cache Protocol (ACP) to dynamically select consistency strategies based on workloads: For tasks with high real-time requirements (such as autonomous driving), a write-broadcast mechanism is used to ensure immediate data updates; for non-real-time tasks (such as non-real-time video streaming), a write-invalidate protocol is used to reduce communication overhead; as shown in the attached Figure 2 shown

[0097] b) Introduce a cache directory table (CCDT) in software to record the cache status and owner information of data blocks, and design a state transition mechanism for different states in hardware. The state transition mechanism implements data consistency management by operating the status and owner fields in CCDT. For example, if the host modifies the shared data block block1, the modification triggers the hardware state transition from valid to modified, and the marking information is

[0098] "Host writes data", then starts executing the broadcast protocol, and the device updates the cache synchronously. The hardware state transition mechanism design is as shown in the attached Figure 3 shown.

[0099] The state machine transition process is described as follows:

[0100] Initialization (init→Invalid): All cache entries are invalid during initialization.

[0101] Data loading (Invalid→Valid): When loading data for the first time, the status changes to Valid

[0102] Data modification (Valid→Modified): write operation triggers state change

[0103] Broadcast update (Modified→Shared): Real-time task triggers write broadcast protocol

[0104] Cache invalidation (Shared→Invalid): Non-real-time tasks use the write invalidation protocol

[0105] Write data: (Shared→Modified): The processor writes data again in the shared state.

[0106] Read data (Valid→Shared): Multiple processors read the same data simultaneously

[0107] Submit data (Modified→Valid): Data is written back to the main memory and submitted.

[0108] In the above, through the custom cache protocol and the hardware state machine transition settings, cache consistency between the host and device sides is achieved. When the host side writes data, the cache status is updated and the device side can obtain it in time.

[0109] The functions of the third module MMU memory management unit are described in detail as follows:

[0110] (1) Address Translation Unit. Responsible for converting virtual addresses to physical addresses. It converts 64-bit virtual addresses into physical addresses through the page table structure and allocates an independent virtual address space for each process.

[0111] (2) Memory allocator. Allocates memory on demand when the task actually needs it, and updates the address mapping changes in real time, supporting fine-grained allocation.

[0112] The design of the fourth module, intelligent data migration module, is described as follows:

[0113] This strategy dynamically tracks memory access patterns and migrates frequently accessed data to the most appropriate processing unit memory based on workload characteristics, reducing memory access latency. Predictive models are used to migrate data in advance to avoid unnecessary memory pauses. Combined with cache-aware data placement and scheduling, this reduces cache evictions, improves cache hit rates, optimizes bandwidth consumption, and ultimately improves system processing efficiency. The design architecture of intelligent data migration technology consists of the following key components, as shown in the attached figure. Figure 4 As shown, the module functions are:

[0114] (1) Data monitoring module: responsible for real-time monitoring of memory access on the host and device sides, and collecting data access frequency, access sequence, spatial locality and other information through hardware performance counters (PMCs) and software probes.

[0115] (2) Analysis and prediction module: Analyze the data collected by the monitoring module and establish a prediction model.

[0116] (3) Decision-making module: Based on the results of the analysis and prediction module, it determines whether data migration is necessary and determines the target location of the migration.

[0117] (4) Migration execution module: This module uses the DMA engine to migrate data from the source location to the target location. This module needs to cooperate with the system's memory management unit (MMU) and cache system to ensure the correctness and efficiency of data migration.

[0118] (5) Information update module: After the data migration is completed, the system memory mapping table, cache directory and other information are updated to ensure that the system can correctly access the migrated data.

[0119] The calculation process of the decision execution part of the intelligent migration module is as follows. Among them, the data migration probability P migrate The calculation formula of is shown in formula ①, where α and β are weight coefficients, whose value range is between ([0,1]) and satisfy α+β=1, which is used to adjust the importance of access frequency and spatial locality in migration decision.

[0120] acc_fre represents the access frequency of data, which is measured by counting the number of times the data is accessed within a specific time period. spatial_locality represents the spatial locality of data, that is, the distribution of data in memory (for example, if the array is stored continuously in memory, the spatial locality is relatively friendly). f(acc_fre) and g(spatial_locality) are two mapping functions, which are used to map the access frequency and spatial locality to the interval ([0,1]). When P migrate When the set threshold is exceeded, the system will trigger the data migration operation.

[0121] P migrate =α*f(acc_fre)+β*g(spatial_locality)①

[0122] The benefit evaluation formula for data migration is shown in formula ②, where T1 and T2 represent the average access delay of data before and after migration, C represents the cost of data access per unit time, and T migrate Indicates the time required for data movement, which includes the time for DMA (Direct Memory Access) to transfer data and the overhead caused by memory management operations.

[0123] R=(T2-T1)*CT migrate ②

[0124] The data migration priority calculation formula is shown in formula ③. When multiple data blocks in the system require migration, this formula is used to calculate the migration priority of each data block and determine which data block to migrate first. The higher the priority, the higher the priority should be migrated.

[0125] Priority=Pp riority *R / Tp riority ③

[0126] Described through the following implementation use case:

[0127] Scenario description: Edge AI devices collect environmental data in real time. After preprocessing on the host side, the device executes the YOLOv5 target detection algorithm and must complete the decision within 100ms.

[0128] Implementation steps:

[0129] 1. Global address mapping. The MMU assigns virtual address 0x1000 to the data and maps it to physical address 0x200000 (host memory).

[0130] 2. Data preprocessing and cache synchronization: The host modifies the obstacle coordinate data, triggering the cache consistency manager's write broadcast protocol to synchronously update the device cache (status changes from Valid → Modified → Shared).

[0131] 3. Intelligent data migration. The intelligent migration unit detects frequent accesses to address 0x1000 on the device side, calculates the data migration probability using formula ①, and triggers DMA migration to the local memory address 0x400000 on the device side.

[0132] 4. Hypothetical parallel computing is performed on the device side. This reduces data access latency, detection latency, and computation speed.

[0133] The above description is only a preferred embodiment of the present invention and is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A unified memory architecture system based on heterogeneous computing, characterized in that: include: a) Global virtual address space module, which creates a shared virtual memory space for the host and device, and supports dynamic address mapping; b) Cache consistency manager, which includes adaptive protocols and cache directory tables to maintain data consistency; c) MMU memory management unit, which implements dynamic memory allocation, page migration and prefetch optimization; d) Intelligent data migration unit, which dynamically migrates hot data to near-end memory based on prediction models to reduce access latency.

2. The system according to claim 1, wherein: The cache coherence manager's adaptive protocol includes: a) Real-time tasks use a write-broadcast protocol to synchronously update all caches through the on-chip network; b) Non-real-time tasks use a write-invalidate protocol, which only marks other caches as invalid; The cache directory table includes: Record the cache status and owner information of the data block, and design a state transition machine for different states in hardware; the state transition mechanism implements data consistency management by operating the status and owner fields in CCDT.

3. The system according to claim 1, wherein: The MMU memory management unit supports: a) Fine-grained memory allocation and dynamic recycling; b) Memory pressure detection and prefetch optimization based on hardware performance counters; include: The address translation unit is responsible for converting virtual addresses to physical addresses; it converts 64-bit virtual addresses into physical addresses through the page table structure and allocates an independent virtual address space for each process; The memory allocator allocates memory on demand when the task actually needs it, and updates the synchronous address mapping changes in real time, supporting fine-grained allocation.

4. The system according to claim 1, wherein: The intelligent data migration unit specifically includes By dynamically tracking memory access patterns, frequently accessed data is migrated to the most appropriate processing unit memory based on workload characteristics, reducing memory access latency; Use predictive models to migrate data in advance to avoid unnecessary memory pauses; Combined with cache-aware data placement and scheduling, it reduces cache evictions, improves cache hit rates, optimizes bandwidth consumption, and ultimately improves system processing efficiency.

5. The system according to claim 6, characterized in that It includes the following modules: (1) Data monitoring module: responsible for real-time monitoring of memory access on the host and device sides, collecting information on data access frequency, access sequence, and spatial locality through hardware performance counters and software probes; (2) Analysis and prediction module: Analyze the data collected by the monitoring module and establish a prediction model; (3) Decision module: Based on the results of the analysis and prediction module, it determines whether data migration is necessary and determines the target location for migration; (4) Migration execution module: uses the DMA engine to migrate data from the source location to the target location; This module needs to collaborate with the system's memory management unit (MMU) and cache system to ensure the correctness and efficiency of data migration. (5) Information update module: After the data migration is completed, the memory mapping table and cache directory in the system are updated to ensure that the system can correctly access the migrated data.

6. The system according to claim 7, characterized in that The data monitoring module collects CPU / GPU memory access data in real time to provide a basis for intelligent decision-making, including (1) Memory access pattern: Track the spatial locality, temporal locality, and operation type of data access and identify frequently accessed data blocks. (2) Processing unit status: monitor CPU / GPU core utilization, cache hit rate, and task characteristics (3) System resource status: statistics of memory usage and bandwidth usage.

7. The system according to claim 5, characterized in that The analysis and prediction module includes: Data preprocessing, GPU high-frequency data index = access frequency × (1-cache hit rate); A threshold value greater than 100 is considered GPU high-frequency data; Clustering and grouping: Use K-means to classify memory blocks into three categories: GPU core data, CPU control data, and shared data, and identify data locality characteristics.

8. The system according to claim 7, characterized in that Forecasting and decision-making process: Use the random forest algorithm to develop a classification prediction model to determine the optimal storage location for data blocks, i.e., to determine whether the data blocks should be stored in CPU memory, GPU memory, or shared cache. (1) Input feature information is access frequency, data block size, cache hit rate, task type (2) Model training (3) Output the optimal storage location label of the data block, and trigger the migration decision when the confidence level is ≥90%; Decision: Based on the random forest output, the decision rule is: If Random Forest Output == 1 (GPU memory): If the remaining space of GPU memory is greater than the data block size: Trigger DMA engine migration, priority = 1; else: First perform memory recovery, release invalid data blocks, and then migrate elif output == 0 (CPU memory): Prioritize placement in the CPU L3 cache. If a cache hit occurs, migration is skipped. else shared cache: Place it in the CPU-GPU shared memory area and enable cache coherence protocol.

9. The system according to claim 5, characterized in that The decision-making mechanism of the intelligent data migration unit includes: a) Migration probability calculation: b) Migration benefit assessment c) When there are several migration data blocks, use migration priority to sort them.

10. The system according to claim 1, wherein: The workflow is as follows: a) Build a global virtual address space to achieve memory sharing between the host and device; b) Maintaining data consistency through the cache consistency manager; c) dynamically optimize data location using intelligent data migration units; d) On-demand memory allocation and address translation based on MMU.

Images