Access Feature-Aware Heterogeneous Memory Dynamic Data Migration System and Method

Abstract

This disclosure presents an embodiment of a heterogeneous memory dynamic data migration system and method based on access feature awareness. One specific implementation of the system includes: an access feature monitoring unit, a migration decision unit, an address remapping unit, and a migration execution unit, all interconnected. The access feature monitoring unit records read / write operation characteristics of data block groups in high-bandwidth memory and high-bandwidth flash memory. The migration decision unit analyzes the migration requirements of data block groups between high-bandwidth memory and high-bandwidth flash memory and generates migration instructions based on these requirements. The address remapping unit maintains a mapping relationship from a unified address space to physical storage locations and dynamically updates the mapping relationship. The migration execution unit executes the migration operation of data block groups between high-bandwidth memory and high-bandwidth flash memory in response to a received migration instruction. This implementation improves system throughput, extends hardware lifespan, and reduces latency.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a heterogeneous memory dynamic data migration system and method based on access feature awareness. Background Technology

[0002] In inference tasks of memory-intensive applications such as large language models, graphics processing units (GPUs) need to store massive amounts of model parameters and intermediate data (such as key-value caches). High-bandwidth memory (HBM) has extremely high access bandwidth, but its unit capacity cost is high and the capacity per chip is limited; while high-bandwidth flash memory (HBF) can provide larger storage capacity and better cost-effectiveness, but its write endurance is lower and its access latency is higher than that of high-bandwidth memory. Currently, the common approaches in existing memory management schemes are: Pure high-bandwidth memory scheme: increasing the number of GPUs to aggregate the total capacity of more high-bandwidth memory. Static memory scheme (represented by the H³ architecture): integrating high-bandwidth memory and high-bandwidth flash memory through high-speed interconnects (such as Daisy-Chain), and providing a unified address space for GPUs. Read-only data (such as model weights and shared pre-computed KV caches) is stored in high-bandwidth flash memory, while generative data (such as real-time KV caches and intermediate activation values) is stored in high-bandwidth memory. Software-layer caching / prefetching schemes: Data swapping is implemented at the software layer using deep learning frameworks or runtime systems. For example, some key-value (KV) caches can be offloaded to CPU memory or solid-state drives (SSDs) and then loaded back into GPU memory when needed.

[0003] However, when adopting the memory management scheme of the above system, the following technical problems often arise: High-bandwidth resource waste, access latency, and decreased system throughput are all consequences of the static pre-partitioning strategy. Data already designated as "read-only" or "cold" remains permanently stored in the high-bandwidth memory, inefficiently consuming these resources. Meanwhile, frequently accessed data stored in the high-bandwidth flash memory cannot be accessed at a faster pace, increasing overall access latency. This rigid data placement limits batch processing size when the amount of data stored in the high-bandwidth memory becomes too large, leading to a decrease in system throughput. Ultimately, this results in wasted high-bandwidth resources, access latency, and reduced system throughput.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a heterogeneous memory dynamic data migration system based on access feature awareness to solve one or more of the technical problems mentioned in the background section above.

[0007] Some embodiments of this disclosure provide a heterogeneous memory dynamic data migration system based on access feature awareness. The system includes: an access feature monitoring unit, a migration decision unit, an address remapping unit, and a migration execution unit that are interconnected. The access feature monitoring unit, deployed in a graphics processor or high-bandwidth memory substrate, is used to monitor and record the read / write operation characteristics of data block groups in the high-bandwidth memory and high-bandwidth flash memory in real time. The migration decision unit is used to analyze the migration requirements of data block groups between the high-bandwidth memory and the high-bandwidth flash memory based on a preset migration condition set and the read / write operation characteristics, and to generate migration instructions according to the migration requirements. The address remapping unit, deployed in the high-bandwidth memory substrate, is used to maintain a mapping relationship from a unified address space to physical storage locations, and dynamically update the mapping relationship in response to receiving the migration instructions. The migration execution unit is used to execute the migration operation of data block groups between the high-bandwidth memory and the high-bandwidth flash memory in response to receiving the migration instructions.

[0008] The various embodiments of this disclosure have the following beneficial effects: Through the heterogeneous memory dynamic data migration system based on access feature awareness of some embodiments of this disclosure, adaptive optimization management of heterogeneous memory resources is achieved through dynamic data awareness and migration operations coordinated by software and hardware. Specifically, the reasons for high bandwidth resource waste, access latency, and decreased system throughput are as follows: Static pre-partitioning strategies cause data that has been converted to a "read-only" or "cold" state to remain in the aforementioned high-bandwidth memory for a long time, ineffectively occupying high bandwidth resources; while data that may be frequently accessed and stored in the aforementioned high-bandwidth flash memory cannot be accessed in a timely manner, thereby increasing the overall access latency. Due to the rigid data placement, when the amount of data stored in the aforementioned high-bandwidth memory is too large, it will limit the batch processing scale, resulting in a decrease in system throughput. Ultimately, this leads to high bandwidth resource waste, access latency, and a decrease in system throughput. Based on this, some embodiments of the heterogeneous memory dynamic data migration system based on access feature awareness disclosed herein include an access feature monitoring unit, a migration decision unit, an address remapping unit, and a migration execution unit that are interconnected. The access feature monitoring unit is deployed in a graphics processor or a high-bandwidth memory substrate and is used to monitor and record the read / write operation characteristics of data block groups in the high-bandwidth memory and high-bandwidth flash memory in real time. This allows the abstract "hot / cold" state to be quantified into specific hardware-processable signals such as read / write frequencies and timestamps, providing an objective and quantitative data basis for subsequent dynamic decision-making. The migration decision unit is used to analyze the migration requirements of data block groups between the high-bandwidth memory and the high-bandwidth flash memory based on a preset migration condition set and the read / write operation characteristics, and to generate migration instructions according to these requirements. This allows the real-time monitored access features to be combined with preset optimization goals (such as reducing latency and balancing load), automatically determining the optimal storage location for each data block through hardware logic, and generating corresponding control commands. The aforementioned address remapping unit, deployed in the high-bandwidth memory substrate, maintains the mapping relationship from a unified address space to physical storage locations and dynamically updates this mapping relationship upon receiving the aforementioned migration instruction. This allows for seamless updating of the processor-accessible address view after a change in the physical location of data, making the entire data migration process completely transparent to the graphics processing unit (GPU) core performing computations. This avoids interruptions or waiting of computational tasks due to data movement, ensuring system service continuity and low latency. The aforementioned migration execution unit, upon receiving the aforementioned migration instruction, executes the migration operation of data block groups between the high-bandwidth memory and the high-bandwidth flash memory. This translates the scheduling instructions of the migration decision unit into actual physical data movement operations, completing data transmission through a dedicated high-speed hardware channel. It serves as the execution terminal for realizing the dynamic flow of data between heterogeneous storage media.This implementation method achieves the effects of improving system throughput, extending hardware lifespan, and reducing latency. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a schematic diagram of the structure of some embodiments of the heterogeneous memory dynamic data migration system based on access feature awareness according to the present disclosure; Figure 2 These are schematic diagrams of other embodiments of the heterogeneous memory dynamic data migration system based on access feature awareness according to this disclosure. Detailed Implementation

[0011] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0012] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0013] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0014] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0015] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0016] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] Please see Figure 1 , Figure 1 This is a schematic diagram 100 of some embodiments of the heterogeneous memory dynamic data migration system based on access feature awareness disclosed herein. The heterogeneous memory dynamic data migration system based on access feature awareness includes: an access feature monitoring unit 101, a migration decision unit 102, an address remapping unit 103, and a migration execution unit 104.

[0018] The access feature monitoring unit 101 can be a hardware circuit deployed in a graphics processor or a high-bandwidth memory substrate for real-time monitoring and recording of the read and write operation characteristics of data block groups in high-bandwidth memory (HBM) and high-bandwidth flash (HBF).

[0019] In some embodiments, the access feature monitoring unit 101 can be deployed in a graphics processor or a high-bandwidth memory substrate to monitor and record the read / write operation characteristics of data block groups in the high-bandwidth memory and high-bandwidth flash memory in real time. The access feature monitoring unit 101 may include, but is not limited to, at least one of the following: a hardware register set. The hardware register set may include a read counter for counting read accesses per unit time, a write counter for counting write accesses per unit time, and a register for recording the timestamp of the last write operation. The data block group can be obtained by dividing the unified address space of the graphics processor into fixed-size data blocks. As an example, the data block may be 4KB or 64KB, aligned with the storage page size. The storage page may be the smallest read / write unit in a flash memory storage device (such as a solid-state drive (SSD), USB flash drive, or memory card). The high-bandwidth memory substrate may be the core silicon wafer of a high-bandwidth memory chip.

[0020] In some alternative implementations of certain embodiments, the access feature monitoring unit 101 described above can be configured to perform the following steps: Allocate hardware register sets for each data block in the aforementioned data block group.

[0021] The migration decision unit 102 can be a hardware control logic unit that analyzes the migration needs of data block groups between high-bandwidth memory and high-bandwidth flash memory based on a preset migration condition set and read / write operation characteristics, and generates migration instructions according to the migration needs.

[0022] In some embodiments, the migration decision unit 102 can be used to analyze the migration requirements of data block groups between the high-bandwidth memory and the high-bandwidth flash memory based on a preset migration condition set and the read / write operation characteristics, and generate migration instructions according to the migration requirements. The migration decision unit 102 may include, but is not limited to, at least one of the following: a finite state machine (FSM), a decision logic circuit, and a control register. The finite state machine can be a synchronous sequential digital circuit composed of a state register, next-state logic, and output logic. The decision logic circuit can be a comparator that implements migration conditions (such as threshold comparison). The control register can be a configurable register for storing strategy parameters (such as threshold, period). The preset migration condition set includes a first migration condition and a second migration condition. The first migration condition includes: the write operation frequency of the data block is lower than a first preset threshold, the read operation frequency of the data block is higher than a second preset threshold, the write operation frequency of the data block is zero within a preset duration, and the occupancy rate of the high-bandwidth memory is higher than a high-water mark threshold. As an example, the first preset threshold can be 1 time / second. The preset duration can be 10 seconds. The second migration condition includes: the increase in the write operation frequency of data blocks per unit time is greater than a third preset threshold; the read operation frequency of data blocks is higher than a fourth preset threshold; the bandwidth utilization rate of the high-bandwidth flash memory meets a preset bottleneck condition; and the utilization rate of the high-bandwidth memory is lower than a high-water mark threshold. For example, the unit time can be 1 second. The third preset threshold can be 10 times / second^2. The fourth preset threshold can be 1000 times / second. The preset bottleneck condition can be 90%. The high-water mark threshold can be 80%.

[0023] In some optional implementations of certain embodiments, the migration decision unit 102 described above can be configured to perform the following steps: The first step is to analyze the read and write operation characteristics of each data block in the aforementioned data block group according to a preset period, obtaining the feature analysis results for each data block. In practice, the execution entity can count the number of write operations and read operations for each data block within the preset period. Then, the number of write operations and read operations are divided by the preset period to obtain the write operation frequency and read operation frequency for each data block. Finally, the write operation frequency and read operation frequency of each data block are determined as the feature analysis results for each data block. As an example, the preset period can be 100 milliseconds.

[0024] The second step involves analyzing the feature analysis results of each data block in the aforementioned data block group against the first migration condition of the aforementioned preset migration condition set to obtain a first analysis result group. In practice, the execution entity can generate a first analysis result representing "true" for each data block in the aforementioned data block group if all feature analysis results of the data block satisfy the first migration condition of the aforementioned preset migration condition set. If not all feature analysis results of the data block satisfy the first migration condition of the aforementioned preset migration condition set, a first analysis result representing "false" is generated. Finally, the first analysis results of each data block are determined as the first analysis result group. The first analysis result can be a Boolean value. The first analysis result can represent "true" or "false".

[0025] The third step involves generating a first migration instruction, based on the first analysis result group, to migrate the data blocks from the high-bandwidth memory to the high-bandwidth flash memory. In practice, the executing entity can select the data blocks corresponding to the "true" first analysis result group as the first group of data blocks to be migrated. Then, a first migration instruction is generated to migrate the first group of data blocks to be migrated from the high-bandwidth memory to the high-bandwidth flash memory. This first migration instruction may include the following: data block identifier, source address (located in the high-bandwidth memory), and destination address (located in the high-bandwidth flash memory).

[0026] The fourth step involves analyzing the feature analysis results of each data block in the aforementioned data block group against the second migration conditions of the aforementioned preset migration condition set to obtain a second analysis result. In practice, the execution entity can generate a second analysis result representing "true" for each data block in the aforementioned data block group if all feature analysis results of the data block satisfy the second migration conditions of the aforementioned preset migration condition set. If not all feature analysis results of the data block satisfy the second migration conditions of the aforementioned preset migration condition set, a second analysis result representing "false" is generated. Finally, the second analysis results of each data block are determined as a second analysis result group. The second analysis result can be a Boolean value. The second analysis result can represent "true" or "false".

[0027] Fifth, based on the second analysis result described above, a second migration instruction is generated to migrate the data blocks from the high-bandwidth flash memory to the high-bandwidth memory. In practice, the executing entity can select the data blocks corresponding to the second analysis result representing "true" from the first analysis result group as the second group of data blocks to be migrated. Then, a second migration instruction representing the migration of the second group of data blocks to be migrated from the high-bandwidth flash memory to the high-bandwidth memory is generated. The second migration instruction may include the following: data block identifier, source address (located in the high-bandwidth flash memory), and destination address (located in the high-bandwidth memory).

[0028] In addressing the technical problems mentioned above by adopting technical solutions, the application scenario—ultra-long sequence inference tasks (e.g., processing long documents with millions of words or analyzing long videos)—often presents the following technical challenges: statically preset migration decision parameters cannot adapt to dynamically changing large oracle model inference loads, leading to inaccurate data migration decisions, imbalanced utilization of heterogeneous memory resources, and consequently, decreased system throughput and hardware wear. Considering the following requirements for this application scenario: real-time perception of data block access behavior, quantification of data block access behavior characteristics, and adaptive adjustment of migration decision logic, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the migration decision unit 102 described above may be further configured to perform the following steps: The first step is to obtain historical migration operation records. In practice, the aforementioned execution entity can read historical migration operation records from internal storage or the monitoring unit. These historical migration operation records include: migration frequency, migration back frequency, high-bandwidth memory occupancy rate, high-bandwidth flash memory occupancy rate, and high-bandwidth flash memory access latency. The migration frequency can be the total number of all data migration operations (including those from high-bandwidth memory to high-bandwidth flash memory and vice versa) performed by the system within the aforementioned preset period. The migration back frequency can be the number of times a data block is migrated back to high-bandwidth memory after being migrated to high-bandwidth flash memory within the aforementioned preset period. The high-bandwidth flash memory occupancy rate can be the ratio of the allocated capacity in high-bandwidth flash memory to its total capacity. The high-bandwidth flash memory access latency can be the average duration actually measured when the system accesses data located in high-bandwidth flash memory.

[0029] The second step involves determining the frequency of invalid migration operations based on the aforementioned historical migration operation records, and then determining the invalid migration rate based on both the invalid migration operation frequency and the migration frequency, serving as the first performance indicator. The invalid migration operation frequency can be defined as a data block that, after being migrated from the source storage medium to the target storage medium (e.g., from high-bandwidth memory to high-bandwidth flash memory), is subsequently migrated back to the source storage medium within a preset observation window. The invalid migration rate can be the ratio of the invalid migration operation frequency to the migration frequency. The preset observation window can be 100 milliseconds, 500 milliseconds, or 1 second.

[0030] The third step is to determine the difference between the high bandwidth memory occupancy rate and the high water level threshold, which will serve as the second performance indicator.

[0031] The fourth step involves determining the average bandwidth utilization rate of the high-bandwidth flash memory based on the aforementioned preset period and the aforementioned high-bandwidth flash memory utilization rate, as the third performance indicator. In practice, the executing entity can determine the average high-bandwidth flash memory utilization rate over the aforementioned preset period as the third performance indicator.

[0032] The fifth step is to determine the difference between the high-bandwidth flash memory access latency and the baseline latency, which will serve as the fourth performance indicator. The baseline latency can be a preset benchmark latency achievable when accessing the high-bandwidth flash memory.

[0033] The sixth step is to determine the first effect indicator, the second effect indicator, the third effect indicator and the fourth effect indicator as the set of effect indicators.

[0034] Step 7: Based on the aforementioned set of effect indicators and the set of target effect values, generate adjustment instructions for at least one threshold parameter. In practice, the executing entity can subtract the first, second, third, and fourth effect indicators from the aforementioned set of effect indicators from the corresponding target effect values ​​in the aforementioned set of target effect values ​​to obtain a set of differences. Then, the set of differences is queried against a preset adjustment rule mapping table to obtain adjustment instructions for at least one threshold parameter. The at least one threshold parameter includes at least one of the aforementioned first, second, third, and fourth preset thresholds. The adjustment instructions may include the following: a threshold parameter identifier, an adjustment direction, and an adjustment amount. For example, the threshold parameter identifier may be either the first or second preset threshold. The adjustment direction may indicate an increase or decrease. The adjustment amount may be a specific numerical change. The preset adjustment rule mapping table may be a preset lookup table. The preset adjustment rule mapping table defines the mapping relationship between "the degree of deviation of the effect indicator" and "the suggested adjustment amount of the corresponding threshold parameter". For example, the aforementioned preset adjustment rule mapping table may include "For every 1% increase in the invalid migration rate above the target value, the first preset threshold should be increased by 2%". The aforementioned set of target effect values ​​may include a first target effect value, a second target effect value, a third target effect value, and a fourth target effect value. As an example, the first target effect value may be 5%, indicating that the system expects invalid migration operations (i.e., operations where data is migrated out and then migrated back shortly after) to be controlled within 5% of the total number of migrations. The second target effect value may be 0%, indicating that the system expects the actual average utilization rate of high-bandwidth memory to be basically consistent with the preset high-water mark threshold. The third target effect value may be 80%, indicating that the system expects the average bandwidth utilization rate of high-bandwidth flash memory to not exceed 80% of its maximum theoretical bandwidth. The fourth target effect value may be 10%, indicating that the system expects the additional latency introduced by accessing high-bandwidth flash memory (compared to the baseline latency) to not exceed 10% of the baseline latency.

[0035] Step 8: Update the preset migration condition set according to the adjustment instructions to obtain an updated migration condition set. In practice, the executing entity can determine the threshold parameter to be adjusted corresponding to the preset migration condition set based on the threshold parameter identifier in the adjustment instructions, and modify the threshold parameter to be adjusted according to the adjustment direction and adjustment amount (e.g., add adjustment amount to the threshold parameter to be adjusted, subtract adjustment amount from the threshold parameter to be adjusted) to obtain the updated migration condition set. The threshold parameter to be adjusted can be any one of the first preset threshold, the preset duration, the second preset threshold, the high water level threshold, the preset threshold, or the fourth preset threshold.

[0036] The first to eighth steps and related content described above constitute an inventive point of this disclosure, and in conjunction with the aforementioned migration execution unit, solve the technical problem that "statically preset migration decision parameters cannot adapt to dynamically changing inference loads of large oracle models, leading to inaccurate data migration decisions, imbalanced utilization of heterogeneous memory resources, and consequently, decreased system throughput and hardware wear." Factors leading to decreased system throughput and hardware wear are often as follows: statically preset migration decision parameters cannot adapt to dynamically changing inference loads of large oracle models, resulting in inaccurate data migration decisions, imbalanced utilization of heterogeneous memory resources, and consequently, decreased system throughput and hardware wear. Solving these factors can improve system throughput, extend hardware lifespan, and reduce latency. To achieve this effect, firstly, historical migration operation records are obtained. This provides a raw and objective data foundation for subsequent quantitative evaluation and root cause analysis. Secondly, based on the aforementioned historical migration operation records, the frequency of invalid migration operations is determined, and based on the frequency of invalid migration operations and the migration frequency, the invalid migration rate is determined as a first performance indicator. Therefore, the accuracy of migration decisions can be precisely quantified. The invalid migration rate directly reflects the proportion of redundant data transfer operations that reduce system efficiency due to improper parameter settings (such as being too aggressive or conservative). Third, the difference between the high-bandwidth memory occupancy rate and the high-water mark threshold is determined as the second performance indicator. This allows for a quantitative assessment of the space pressure on this expensive resource, high-bandwidth memory, and determines whether the current data distribution is conducive to freeing up sufficient space to support larger-scale processing. Fourth, based on the preset period and the high-bandwidth flash memory occupancy rate, the average bandwidth utilization rate of the high-bandwidth flash memory is determined as the third performance indicator. This allows for a quantitative assessment of the access pressure on the high-bandwidth flash memory and timely detection of whether its bandwidth has become a bottleneck limiting system performance. Fifth, the difference between the high-bandwidth flash memory access latency and the baseline latency is determined as the fourth performance indicator. This allows for a quantitative assessment of the actual performance loss caused by data residing in high-bandwidth flash memory, providing a direct basis for determining whether hot data should be migrated back. Sixth, the first, second, third, and fourth performance indicators are defined as a set of performance indicators. Therefore, a multi-dimensional and comprehensive evaluation system can be constructed to fully characterize the overall effect of the current migration strategy from three aspects: decision accuracy, resource balance, and performance impact. Seventh, based on the above set of effect indicators and target effect values, adjustment instructions for at least one threshold parameter are generated. This allows the comprehensive evaluation results to be transformed into specific, executable control commands. Eighth, according to the above adjustment instructions, the above preset migration condition set is updated to obtain an updated migration condition set. This allows its built-in decision rules to dynamically adapt to the latest workload characteristics.Finally, by combining the aforementioned migration execution units, data migration operations based on dynamic optimization strategies can be implemented, thereby improving system throughput, extending hardware lifespan, and reducing latency.

[0037] In addressing the technical challenges of ultra-long sequence reasoning tasks, the following technical issues often arise when applying solutions to scenarios involving ultra-long content with strong phased access characteristics (e.g., summarizing, answering questions, or coherently continuing sentences from novels, research papers, or legal documents with millions of words). Specifically, during the adaptive adjustment of multiple decision threshold parameters based on multi-dimensional performance metrics, the complex coupling between performance metrics and threshold parameters, coupled with a lack of efficient decision mapping mechanisms, can lead to a large amount of redundant data migration and address remapping operations, increasing bandwidth consumption and access latency in heterogeneous memory channels. Considering the following requirements for this application scenario: clear causal mapping relationships, quantification of biases, and short decision-making processes, we have decided to adopt the following solution: In some alternative implementations of certain embodiments, the generation of adjustment instructions for at least one threshold parameter based on the aforementioned set of effect indicators and set of target effect values ​​includes: The first step is to determine the difference between each effect indicator in the aforementioned set of effect indicators and the corresponding target effect value in the aforementioned set of target effect values, thus obtaining a difference set. In practice, the implementing entity can determine the difference between the aforementioned first effect indicator and the aforementioned first target effect value as the first difference. The difference between the aforementioned second effect indicator and the aforementioned second target effect value is determined as the second difference. The difference between the aforementioned third effect indicator and the aforementioned third target effect value is determined as the third difference. The difference between the aforementioned fourth effect indicator and the aforementioned fourth target effect value is determined as the fourth difference. Then, the aforementioned first difference, second difference, third difference, and fourth difference are merged into a difference set.

[0038] The second step involves performing the following root cause correlation analysis on the aforementioned set of differences and the preset migration conditions: Step one: In response to a positive difference value corresponding to the first effect indicator, the first preset threshold and the preset duration are determined as the first set of threshold parameters to be adjusted. In practice, the executing entity can determine the first preset threshold and the preset duration as the first set of threshold parameters to be adjusted in response to a positive first difference value. As an example, a positive first difference value indicates that the system has performed too many "invalid migration" operations, requiring adjustment of the first preset threshold and the preset duration.

[0039] Step two: In response to a positive difference value corresponding to the second effect indicator, the second preset threshold and the high watermark threshold are determined as the second set of threshold parameters to be adjusted. In practice, the executing entity can determine the second preset threshold and the high watermark threshold as the second set of threshold parameters to be adjusted in response to a positive second difference value. As an example, a positive second difference value indicates that the high-bandwidth memory is experiencing continuous space shortage, requiring adjustment of the second preset threshold and the high watermark threshold.

[0040] Step three: In response to a positive difference between the third or fourth performance indicator, the third and fourth preset thresholds are determined as the third set of threshold parameters to be adjusted. In practice, the executing entity may determine the third and fourth preset thresholds as the third set of threshold parameters to be adjusted in response to a positive third or fourth difference. As an example, a positive third or fourth difference indicates a performance bottleneck in the high-bandwidth flash memory, requiring adjustment of the third and fourth preset thresholds.

[0041] Step four: The first set of threshold parameters to be adjusted, the second set of threshold parameters to be adjusted, and the third set of threshold parameters to be adjusted are determined as the set of threshold parameters to be adjusted.

[0042] The third step involves querying a preset adjustment rule mapping table based on the aforementioned set of differences and the aforementioned set of threshold parameters to be adjusted, in order to generate an adjustment scheme for the aforementioned set of threshold parameters to be adjusted. In practice, the executing entity can use the aforementioned set of differences and the aforementioned set of threshold parameters to be adjusted as input to query the aforementioned preset adjustment rule mapping table to generate an adjustment scheme for the aforementioned set of threshold parameters to be adjusted.

[0043] The fourth step involves generating adjustment instructions for at least one threshold parameter based on the aforementioned adjustment scheme. In practice, the executing entity can parse the adjustment scheme to extract each adjustment parameter. Then, these adjustment parameters are encoded into a binary command format to obtain the adjustment instructions for at least one threshold parameter. For example, the adjustment parameters could be [threshold parameter identifier, adjustment direction, adjustment amount]. The adjustment instructions include the threshold parameter identifier, adjustment direction, and adjustment amount.

[0044] The first to fourth steps and related content described above constitute an inventive point of this disclosure. Combined with the aforementioned migration execution unit, they enable data migration operations based on a dynamic optimization strategy, solving the technical problem that "in the process of adaptively adjusting multiple decision threshold parameters based on multi-dimensional performance indicators, due to the complex coupling relationship between performance indicators and threshold parameters and the lack of an efficient decision mapping mechanism, the adaptive adjustment process may generate a large number of redundant data migration and address remapping operations, thereby increasing the bandwidth consumption and access latency of heterogeneous memory channels." Factors leading to increased bandwidth consumption and access latency often include: in the process of adaptively adjusting multiple decision threshold parameters based on multi-dimensional performance indicators, due to the complex coupling relationship between performance indicators and threshold parameters and the lack of an efficient decision mapping mechanism, the adaptive adjustment process may generate a large number of redundant data migration and address remapping operations, thereby increasing the bandwidth consumption and access latency of heterogeneous memory channels. If these factors are resolved, the effect of reducing bandwidth consumption and access latency can be achieved. To achieve this effect, firstly, the difference between each performance indicator in the aforementioned performance indicator set and the corresponding target performance value in the target performance value set is determined, resulting in a difference set. Therefore, the multi-dimensional and comprehensive effect evaluation results can be transformed into a series of calculable, directional (positive / negative) quantitative deviations. Second, based on the above set of differences, the following root cause correlation analysis is performed on the above preset migration condition set: Step one, in response to the positive difference value corresponding to the first effect indicator, the above first preset threshold and the above preset duration are determined as the first set of threshold parameters to be adjusted. Thus, the decision accuracy problem of "high invalid migration rate" can be precisely located as potentially caused by overly sensitive conditions for judging data "cooling" (write frequency threshold, write cooling time), clarifying the primary optimization direction. Step two, in response to the positive difference value corresponding to the second effect indicator, the above second preset threshold and the above high water level threshold are determined as the second set of threshold parameters to be adjusted. Thus, the resource pressure problem of "insufficient high-bandwidth memory space release" can be associated with potentially overly conservative conditions for migrating data from high-bandwidth memory (read frequency threshold, space warning line), indicating the parameter adjustment path for releasing resources. Step 3: Responding to a positive difference between the third or fourth effect metric, the third and fourth preset thresholds are determined as the third set of threshold parameters to be adjusted. This links the issue of "performance bottlenecks in high-bandwidth flash memory" to the possibility that the conditions for migrating hot data back to high-bandwidth memory (write mutation threshold, high read frequency threshold) are too lenient, thus identifying the key adjustment parameters for improving performance. Step 4: The first, second, and third sets of threshold parameters to be adjusted are determined as the set of threshold parameters to be adjusted. This allows for the integration of all root cause analysis results, forming a complete list of parameters to be optimized.Third, based on the aforementioned set of differences and the set of threshold parameters to be adjusted, a preset adjustment rule mapping table is queried to generate an adjustment scheme for the set of threshold parameters to be adjusted. This transforms the qualitative judgment of "which parameters need to be adjusted" into a deterministic operational scheme of "how much to adjust each parameter." Fourth, based on the aforementioned adjustment scheme, an adjustment instruction for at least one threshold parameter is generated. This allows the adjustment scheme to be encoded into a low-level control command that the migration decision unit can directly recognize and execute, driving the decision engine to complete real-time updates of its internal strategy parameters, thereby achieving strategy iteration. Finally, combined with the aforementioned migration execution unit, a data migration operation based on a dynamically optimized strategy is realized. This reduces bandwidth consumption and access latency.

[0045] The address remapping unit 103 may be an address management hardware deployed in the substrate of a high-bandwidth memory to maintain the mapping relationship from a unified address space to a physical storage location and to dynamically update the mapping relationship in response to a migration instruction.

[0046] In some embodiments, the address remapping unit 103 can be deployed in the high-bandwidth memory substrate to maintain the mapping relationship from a unified address space to physical storage locations, and dynamically update the mapping relationship in response to receiving the migration instruction. The address remapping unit 103 may include, but is not limited to, at least one of the following: a page table memory and a Translation Lookaside Buffer (TLB). The page table memory may be a static random-access memory (SRAM) used to store a mapping table of "virtual page number - physical storage type - physical page number". In some optional implementations of certain embodiments, the address remapping unit 103 described above can be configured to perform the following steps: The first step is to set up and update a page mapping table in the static random access memory (SRAM) of the high-bandwidth memory substrate. In practice, the execution entity can create and initialize a page mapping table in the SRAM located on the high-bandwidth memory substrate during system initialization. During system operation, the page mapping table is updated in real time. This page mapping table can be a hardware data structure that records the mapping relationship between virtual page addresses in the unified address space and physical memory types and physical page addresses.

[0047] The second step involves determining and storing the new mapping relationship in response to receiving the mapping relationship update instruction from the aforementioned migration execution unit. In practice, the execution entity can, in response to receiving the mapping relationship update instruction from the aforementioned migration execution unit, determine the mapping relationship update instruction as the new mapping relationship and store the new mapping relationship in a dedicated buffer. The mapping relationship update instruction contains the virtual address of the data block and the target physical location. The new mapping relationship can be a mapping from the virtual address of the data block to the target physical location.

[0048] Third, in response to receiving the mapping relationship activation instruction from the aforementioned migration execution unit, the page mapping table is updated through an atomic operation and the new mapping relationship. In practice, the aforementioned execution entity can, in response to receiving the mapping relationship activation instruction from the aforementioned migration execution unit, perform an atomic operation: replacing the old mapping relationship for the corresponding virtual address in the aforementioned page mapping table with the new mapping relationship in the aforementioned dedicated buffer.

[0049] The migration execution unit 104 may be background data transfer hardware for performing migration operations of data block groups between high-bandwidth memory and high-bandwidth flash memory in response to receiving a migration instruction.

[0050] In some embodiments, the migration execution unit 104 is configured to perform a migration operation of the data block group between the high-bandwidth memory and the high-bandwidth flash memory in response to receiving the migration instruction. The migration execution unit 104 may include, but is not limited to, at least one of the following: a memory access controller, a data buffer, and a channel controller.

[0051] In some optional implementations of certain embodiments, the migration execution unit 104 described above can be configured to perform the following steps: The first step is to lock the data block upon receiving a first or second migration instruction from the aforementioned migration decision unit. In practice, the execution entity can perform a locking operation on the data block corresponding to the data block identifier upon receiving the first or second migration instruction from the aforementioned migration decision unit. This locking operation can be a hardware-implemented access control operation. As an example, the locking operation can temporarily prohibit write access to the data block by setting a hardware status bit (such as an "occupied" flag) associated with the data block.

[0052] The second step is to send a mapping relationship update instruction to the aforementioned address remapping unit.

[0053] The third step involves copying the data block to the target storage medium via the high-speed inter-chip interconnect channel between the high-bandwidth memory and the high-bandwidth flash memory. In practice, the execution entity can copy the data block to the target storage medium (high-bandwidth flash memory or high-bandwidth memory) via the high-speed inter-chip interconnect channel between the high-bandwidth memory and the high-bandwidth flash memory. This high-speed inter-chip interconnect channel can be a dedicated high-speed physical interface for connecting two chips (such as a high-bandwidth memory substrate and a high-bandwidth flash memory controller). This high-speed inter-chip interconnect channel can provide high-bandwidth, low-latency data transmission capabilities. For example, the high-speed inter-chip interconnect channel can be a die-to-die (D2D) interface. Figure 2 As shown in the structural schematic diagrams of other embodiments of the heterogeneous memory dynamic data migration system based on access feature awareness disclosed herein, the GPU (Graphics Processing Unit) / TPU (Tensor Processing Unit) / XPU (Extended Processing Unit) can be the computing core. The HBM Core can be the storage core of a high-bandwidth memory; for example, each HBM Core can be an independent HBM stack integrating DRAM storage cells. The HBM Base can be the aforementioned high-bandwidth memory substrate. The HBFCore can be the storage core of a high-bandwidth flash memory, for example, a flash memory-based storage medium (such as 3D NAND). The HBF Base can be the controller substrate of a high-bandwidth flash memory; for example, the aforementioned HBF Base can integrate a flash memory controller, etc. The Interposer can be a large passive silicon wafer located beneath all active chips. D2D can be an ultra-short distance, ultra-high bandwidth physical trace etched onto the aforementioned Interposer.

[0054] The fourth step is to send a mapping relationship activation instruction to the aforementioned address remapping unit.

[0055] The fifth step is to release the storage space occupied by the data block. In practice, the execution entity can cancel the above execution locking operation and mark the physical storage space occupied by the data block in the source storage medium (high-bandwidth memory or high-bandwidth flash memory) as free and available.

[0056] The various embodiments of this disclosure have the following beneficial effects: Through the heterogeneous memory dynamic data migration system based on access feature awareness of some embodiments of this disclosure, adaptive optimization management of heterogeneous memory resources is achieved through dynamic data awareness and migration operations coordinated by software and hardware. Specifically, the reasons for high bandwidth resource waste, access latency, and decreased system throughput are as follows: Static pre-partitioning strategies cause data that has been converted to a "read-only" or "cold" state to remain in the aforementioned high-bandwidth memory for a long time, ineffectively occupying high bandwidth resources; while data that may be frequently accessed and stored in the aforementioned high-bandwidth flash memory cannot be accessed in a timely manner, thereby increasing the overall access latency. Due to the rigid data placement, when the amount of data stored in the aforementioned high-bandwidth memory is too large, it will limit the batch processing scale, resulting in a decrease in system throughput. Ultimately, this leads to high bandwidth resource waste, access latency, and a decrease in system throughput. Based on this, some embodiments of the heterogeneous memory dynamic data migration system based on access feature awareness disclosed herein include an access feature monitoring unit, a migration decision unit, an address remapping unit, and a migration execution unit that are interconnected. The access feature monitoring unit is deployed in a graphics processor or a high-bandwidth memory substrate and is used to monitor and record the read / write operation characteristics of data block groups in the high-bandwidth memory and high-bandwidth flash memory in real time. This allows the abstract "hot / cold" state to be quantified into specific hardware-processable signals such as read / write frequencies and timestamps, providing an objective and quantitative data basis for subsequent dynamic decision-making. The migration decision unit is used to analyze the migration requirements of data block groups between the high-bandwidth memory and the high-bandwidth flash memory based on a preset migration condition set and the read / write operation characteristics, and to generate migration instructions according to these requirements. This allows the real-time monitored access features to be combined with preset optimization goals (such as reducing latency and balancing load), automatically determining the optimal storage location for each data block through hardware logic, and generating corresponding control commands. The aforementioned address remapping unit, deployed in the high-bandwidth memory substrate, maintains the mapping relationship from a unified address space to physical storage locations and dynamically updates this mapping relationship upon receiving the aforementioned migration instruction. This allows for seamless updating of the processor-accessible address view after a change in the physical location of data, making the entire data migration process completely transparent to the graphics processing unit (GPU) core performing computations. This avoids interruptions or waiting of computational tasks due to data movement, ensuring system service continuity and low latency. The aforementioned migration execution unit, upon receiving the aforementioned migration instruction, executes the migration operation of data block groups between the high-bandwidth memory and the high-bandwidth flash memory. This translates the scheduling instructions of the migration decision unit into actual physical data movement operations, completing data transmission through a dedicated high-speed hardware channel. It serves as the execution terminal for realizing the dynamic flow of data between heterogeneous storage media.This implementation method achieves the effects of improving system throughput, extending hardware lifespan, and reducing latency.

[0057] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A heterogeneous memory dynamic data migration system based on access feature awareness, wherein, The system includes: an access feature monitoring unit, a migration decision unit, an address remapping unit, and a migration execution unit that are interconnected. Access feature monitoring unit, deployed in graphics processor or high-bandwidth memory substrate, is used to monitor and record the read and write operation features of data block groups in the high-bandwidth memory and high-bandwidth flash memory in real time; The migration decision unit is used to analyze the migration requirements of the data block group between the high-bandwidth memory and the high-bandwidth flash memory based on a preset migration condition set and the read / write operation characteristics, and generate migration instructions according to the migration requirements; An address remapping unit, deployed in the high-bandwidth memory substrate, is used to maintain the mapping relationship from a unified address space to physical storage locations, and dynamically updates the mapping relationship in response to receiving the migration instruction; A migration execution unit is configured to, in response to receiving the migration instruction, perform a migration operation of the data block group between the high-bandwidth memory and the high-bandwidth flash memory.

2. The system according to claim 1, wherein, The access feature monitoring unit is configured to: A hardware register set is allocated to each data block in the data block group, wherein the hardware register set includes a read counter for counting the number of read accesses per unit time, a write counter for counting the number of write accesses per unit time, and a register for recording the timestamp of the last write operation.

3. The system according to claim 1, wherein, The address remapping unit is configured to: In the static random access memory of the high-bandwidth memory substrate, a page mapping table is set and updated, wherein the page mapping table is used to record the mapping relationship between virtual page addresses in the unified address space and physical memory types and physical page addresses; In response to receiving a mapping update instruction from the migration execution unit, a new mapping relationship is determined and stored; In response to receiving a mapping relationship activation instruction from the migration execution unit, the page mapping table is updated through atomic operations and the new mapping relationship.

4. The system according to claim 1, wherein, The migration execution unit is configured to: In response to receiving a first migration instruction or a second migration instruction from the migration decision unit, the data block is locked; Send a mapping update instruction to the address remapping unit; Data blocks are copied to the target storage medium via the high-speed inter-chip interconnect channel between the high-bandwidth memory and the high-bandwidth flash memory. Send a mapping relationship activation instruction to the address remapping unit; Release the storage space occupied by the data block.

5. The system according to claim 1, wherein, The migration decision unit is configured to: According to a preset period, the read and write operation characteristics of each data block in the data block group are analyzed to obtain the feature analysis results of each data block; For each data block in the data block group, the feature analysis results of the data block are analyzed with the first migration condition of the preset migration condition set to obtain a first analysis result group. The first migration condition includes: the write operation frequency of the data block is lower than a first preset threshold, the read operation frequency of the data block is higher than a second preset threshold, the write operation frequency of the data block is zero within a preset time period, and the occupancy rate of the high bandwidth memory is higher than the high water level threshold. Based on the first set of analysis results, a first migration instruction is generated to migrate the data block from the high-bandwidth memory to the high-bandwidth flash memory; For each data block in the data block group, the feature analysis results of the data block are analyzed with the second migration conditions of the preset migration condition set to obtain a second analysis result. The second migration conditions include: the increase value of the write operation frequency of the data block in a unit time is greater than a third preset threshold, the read operation frequency of the data block is higher than a fourth preset threshold, the bandwidth utilization rate of the high bandwidth flash memory meets the preset bottleneck condition, and the utilization rate of the high bandwidth memory is lower than the high water level threshold. Based on the second analysis result, a second migration instruction is generated to migrate the data block from the high-bandwidth flash memory to the high-bandwidth storage.

Images