Data interaction method, device, equipment and storage medium for multi-layer stacked memory

Abstract

The present invention relates to a data interaction method, device, equipment and storage medium of a multi-layer stacked memory, comprising the following steps: based on the access frequency clustering result, the target transmission data is mapped to a storage location to obtain a multi-layer storage allocation strategy table; based on the multi-layer storage allocation strategy table, the target transmission data is interactively planned to obtain an optimal interactive path plan; based on the optimal interactive path plan, the target transmission data is hierarchically migrated and scheduled to obtain a data migration execution queue; and the data migration execution queue is transmitted in parallel through an asynchronous pipeline transmission mechanism, which solves the technical problem that the existing data transmission strategies often lack in-depth analysis of the historical access characteristics of the storage hierarchy, resulting in unreasonable data distribution, and frequent cross-level data migration increases system delay and energy consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-layer stacked memory, and in particular to a data interaction method, device, equipment and storage medium for a multi-layer stacked memory. Background Art

[0002] With the rapid development of information technology, the demand for data storage is growing. The traditional single-layer memory architecture is gradually unable to cope with the needs of massive data and fast access. In order to improve storage efficiency and response speed, multi-layer stacked memory has emerged as an innovative solution. It builds a hierarchical storage structure by integrating different types of storage media, such as cache, dynamic random access memory (DRAM), flash memory, and hard disk. This structure can intelligently allocate data according to access frequency and characteristics, thereby optimizing read and write performance and power management. However, in practical applications, how to achieve efficient data interaction has become a major challenge.

[0003] One of the problems in the research background is that traditional data interaction methods fail to fully utilize the advantages of multi-layer stacked memory. Existing data transmission strategies often lack in-depth analysis of the historical access characteristics of the storage hierarchy, resulting in unreasonable data distribution, and frequent cross-level data migration increases system latency and energy consumption. In addition, due to the lack of effective clustering algorithms to identify and classify data access patterns, it is difficult for the system to formulate an optimal storage location mapping scheme, which further affects the overall performance. Therefore, a new data interaction method is urgently needed to overcome these limitations to ensure that data can be efficiently processed at the appropriate storage level.

[0004] In response to the above problems, a data interaction method based on frequency clustering analysis and bidirectional hash mapping algorithm is proposed to improve the working efficiency of multi-layer stacked memory. This method not only takes into account the historical access characteristic parameters of the target memory, but also introduces advanced clustering technology to analyze the access pattern of the data, so that the data can be accurately classified according to its access frequency. In this way, personalized storage allocation strategies can be formulated for different types of data, and then the optimal interaction path can be planned. Finally, parallel data transmission is achieved with the help of asynchronous pipe transmission mechanism, which significantly reduces the time cost of data migration, improves the system's response speed and energy efficiency, and meets the urgent needs of modern information processing for high-performance storage systems. Summary of the invention

[0005] The main purpose of the present invention is to provide a data interaction method, device, equipment and storage medium for a multi-layer stacked memory, which solves the technical problems that existing data transmission strategies often lack in-depth analysis of historical access characteristics of storage levels, resulting in unreasonable data distribution and frequent cross-level data migration, which increases system latency and energy consumption.

[0006] To achieve the above object, the present invention provides a data interaction method of a multi-layer stacked memory, comprising the following steps:

[0007] Obtaining historical access characteristic parameters of the storage level in the target memory and target transmission data input into the target memory, and performing frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein the multi-layer stacked memory is used as the target memory;

[0008] By using a bidirectional hash mapping algorithm, the target transmission data is mapped to a storage location based on the access frequency clustering result to obtain a multi-layer storage allocation strategy table;

[0009] Performing interactive path planning on the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning;

[0010] Based on the optimal interaction path planning, hierarchical migration scheduling is performed on the target transmission data to obtain a data migration execution queue;

[0011] The data migration execution queue is subjected to parallel data transmission via an asynchronous pipeline transmission mechanism.

[0012] Further, the frequency clustering analysis is performed on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results, including:

[0013] Dividing the historical access characteristic parameters into time windows to obtain a dynamic time window sequence;

[0014] Performing data access frequency analysis on the dynamic time window sequence by using a maximum entropy clustering algorithm to obtain a data access frequency distribution diagram;

[0015] Based on the data access frequency distribution diagram, the historical access characteristic parameters are divided into intervals to obtain a frequency interval division result; wherein the frequency interval division result includes a high-frequency access interval, a medium-frequency access interval and a low-frequency access interval;

[0016] Based on the frequency interval division result, frequency clustering analysis is performed on the target transmission data to obtain access frequency clustering results; wherein the access frequency clustering results include hot data, warm data and cold data.

[0017] Furthermore, the target transmission data is mapped to a storage location based on the access frequency clustering result by a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table, including:

[0018] Performing location-sensitive hash coding on the access frequency clustering results to obtain an initial hash mapping table, and performing consistent hash ring construction on the initial hash mapping table to obtain a ring address space; wherein the ring address space includes virtual node distribution identifiers, physical address mapping relationships, and conflict area markers;

[0019] Performing conflict detection on the annular address space through a Bloom filter to obtain a conflict detection result, and constructing a skip table index on the conflict detection result to obtain a multi-level index structure; wherein the multi-level index structure includes a hot data index layer, a warm data index layer, and a cold data index layer;

[0020] If a hash bucket overflow occurs in the load factor of any layer of the multi-level index structure, the multi-level index structure is dynamically bucket-split by an extensible hash algorithm to obtain a hash bucket sequence, and a load balancing analysis is performed on the hash bucket sequence to obtain balancing factor data; wherein the balancing factor data includes storage capacity distribution, access load distribution and bandwidth utilization;

[0021] The balancing factor data is collaboratively optimized to obtain an optimized weight vector, and a storage location mapping is performed on the target transmission data based on the optimized weight vector through a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table.

[0022] Furthermore, performing interactive path planning on the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning includes:

[0023] Based on the multi-layer storage allocation strategy table, an initial path is constructed for the target transmission data to obtain a set of candidate paths, and a path delay prediction is performed on the set of candidate paths to obtain a path delay prediction table; wherein the path delay prediction table includes an estimated transmission delay, an estimated queuing delay, and an estimated congestion delay of each candidate path;

[0024] Performing path search and optimization on the path delay prediction table by using an ant colony optimization algorithm to obtain a path efficiency ranking list, and performing energy consumption balance analysis on the path efficiency ranking list to obtain an energy consumption balance evaluation table; wherein the energy consumption balance evaluation table includes the energy consumption level of each candidate path and the energy consumption distribution of the storage level;

[0025] Pruning the path nodes in the path efficiency ranking list based on the energy consumption balance evaluation table to obtain a pruned path set;

[0026] Performing path topology optimization on the pruned path set by a genetic algorithm to obtain an optimized path topology structure, and performing bandwidth resource allocation on the optimized path topology structure to obtain a bandwidth resource allocation table; wherein the bandwidth resource allocation table includes bandwidth allocation conditions and bandwidth utilization efficiency of each path segment;

[0027] Based on the bandwidth resource allocation table, QoS policy configuration is performed on the optimized path topology structure to obtain a QoS policy configuration table, and path stability evaluation is performed on the QoS policy configuration table to obtain a path stability report; wherein the QoS policy configuration table includes a service quality level, priority setting, and fault tolerance mechanism for each path segment;

[0028] The optimal path is selected for the path stability report through a dynamic programming algorithm to obtain an optimal interactive path planning; wherein the optimal interactive path planning includes a selected optimal path and a backup path.

[0029] Furthermore, performing hierarchical migration scheduling on the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue includes:

[0030] Based on the optimal interactive path planning, the target transmission data is divided into blocks to obtain a data block set, and the data block set is prioritized to obtain a priority queue;

[0031] Allocating time slices to the priority queues by a time slice round-robin algorithm to obtain a time slice allocation table, and performing resource competition prediction on the time slice allocation table to obtain a resource competition prediction graph; wherein the resource competition prediction graph includes resource requirements, potential resource conflicts, and competition levels of each of the data block sets;

[0032] The scheduling strategy of the time slice allocation table is adjusted based on the resource competition prediction graph to obtain an adjusted time slice allocation table, and data migration dependency analysis is performed on the adjusted time slice allocation table to obtain a data migration dependency graph; wherein the data migration dependency graph is used to describe the dependency and sequence between the data block sets during the migration process;

[0033] Sorting the data migration dependency graph in migration order by using a topological sorting algorithm to obtain a data migration order list;

[0034] The data migration order list is encapsulated as a data migration task to obtain a data migration task queue, and the data migration task queue is hierarchically scheduled through a multi-level feedback queue scheduling algorithm to obtain a data migration execution queue.

[0035] Furthermore, the performing parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism includes:

[0036] Dividing the data migration execution queue into pipeline segments to obtain a pipeline segment sequence, and calculating a flow control threshold value for the pipeline segment sequence to obtain a flow control parameter set;

[0037] Performing traffic shaping processing on the flow control parameter set through a preset token bucket algorithm to obtain a transmission rate control table, and dynamically allocating a buffer zone on the transmission rate control table to obtain a buffer zone configuration scheme;

[0038] Based on the buffer configuration scheme, a parallel transmission channel is constructed for the pipeline segment sequence to obtain a parallel transmission topology map, and a semaphore synchronization analysis is performed on the parallel transmission topology map to obtain a synchronization control strategy; wherein the synchronization control strategy includes mutually exclusive access control, synchronization signal triggering conditions and deadlock avoidance mechanism;

[0039] The synchronous control strategy is optimized for data transmission by using zero-copy technology to obtain an optimized transmission scheme, and the optimized transmission scheme is scheduled for transmission timing to obtain a pipeline transmission scheduling table;

[0040] Through a preset asynchronous pipeline transmission mechanism, parallel data transmission is performed on the data migration execution queue based on the pipeline transmission scheduling table.

[0041] Further, the flow control parameter set is subjected to flow shaping processing by a preset token bucket algorithm to obtain a transmission rate control table, including:

[0042] Calculating the token generation rate for the flow control parameter set to obtain a token generation periodic table;

[0043] The bucket capacity of the token generation periodic table is calculated by a dynamic bucket depth adjustment algorithm to obtain a token bucket depth distribution diagram;

[0044] Performing burst traffic processing on the token bucket depth distribution graph to obtain a burst traffic control strategy, which includes a maximum burst capacity, a burst traffic buffer size, and a traffic smoothing factor;

[0045] The burst flow control strategy is allocated tokens by a preset token bucket algorithm to obtain token allocation weight data; wherein the token allocation weight data includes priority weight, delay sensitivity and throughput requirement;

[0046] Performing traffic shaping processing on the target transmission data based on the token allocation weight data to obtain a traffic shaping parameter set;

[0047] The traffic shaping parameter set is rate constrained by a layered rate limiting algorithm to obtain a transmission rate control table; wherein the transmission rate control table includes rate limits for each layer, rate ratios between layers, and rate adjustment steps.

[0048] The present invention also provides a data interaction device for a multi-layer stacked memory, comprising:

[0049] an acquisition module, used to acquire historical access characteristic parameters of storage layers in a target memory and target transmission data input into the target memory, and to perform frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein the multi-layer stacked memory is used as the target memory;

[0050] A mapping module, used to perform storage location mapping on the target transmission data based on the access frequency clustering result by a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table;

[0051] A planning module, used to perform interactive path planning for the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning;

[0052] A scheduling module, configured to perform hierarchical migration scheduling on the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue;

[0053] The transmission module is used to perform parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism.

[0054] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0055] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0056] The data interaction method of a multi-layer stacked memory provided by the present invention comprises the following steps: obtaining historical access characteristic parameters of a storage layer in a target memory and inputting target transmission data of the target memory, and performing frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein, the multi-layer stacked memory is used as the target memory; using a bidirectional hash mapping algorithm, based on the access frequency clustering results, storage location mapping is performed on the target transmission data to obtain a multi-layer storage allocation strategy table; based on the multi-layer storage allocation strategy table, interaction path planning is performed on the target transmission data to obtain an optimal interaction path planning; based on the optimal interaction path planning, hierarchical migration scheduling is performed on the target transmission data to obtain a data migration execution queue; and parallel data transmission is performed on the data migration execution queue through an asynchronous pipeline transmission mechanism, thereby solving the technical problems that the existing data transmission strategies often lack in-depth analysis of the historical access characteristics of the storage layer, resulting in unreasonable data distribution, and frequent cross-layer data migration increases system delay and energy consumption, and realizes the use of a bidirectional hash mapping algorithm to perform storage location mapping on the data after frequency clustering analysis to generate a multi-layer storage allocation strategy table. This method not only ensures that data can be reasonably distributed on different storage levels according to its access characteristics, but also avoids the resource waste problem that may be caused by traditional static allocation, and effectively improves the technical effect of storage resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a schematic diagram of the steps of a data interaction method of a multi-layer stacked memory in one embodiment of the present invention;

[0058] Figure 2 is a structural block diagram of a data interaction device of a multi-layer stacked memory in one embodiment of the present invention;

[0059] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0060] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] like Figure 1 As shown, Figure 1 This is a schematic diagram of the steps of a data interaction method for a multi-layer stacked memory in one embodiment of the present invention;

[0063] In one embodiment of the present invention, a data interaction method of a multi-layer stacked memory is provided, comprising the following steps:

[0064] Step S1, obtain the historical access characteristic parameters of the storage layer in the target memory and the target transmission data input into the target memory, and perform frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein the multi-layer stacked memory is used as the target memory.

[0065] Specifically, when implementing the steps described above, it is first necessary to obtain historical access characteristic parameters of each storage layer from the target storage. These parameters are a collection of data that reflects the access frequency and pattern of different layers in the past period of time. For example, some data blocks may be frequently read or written, while others are rarely touched. At the same time, the system also receives target transmission data input to the target storage, which may be new or data that already exists in the storage but needs to be relocated. Once the historical access characteristic parameters and target transmission data are collected, the next step is to perform frequency clustering analysis on the target transmission data based on these historical access characteristic parameters. In this process, the algorithm will classify data with similar access frequencies together according to historical access patterns, thereby obtaining access frequency clustering results. For example, in a multi-layer stacked storage environment, if a group of data is found to have been frequently requested in the past, then this group of data will be marked as hot data; on the contrary, data that has not been accessed for a long time will be considered cold data. This classification is crucial for the subsequent decision on which storage layer the data should be stored, because different storage layers have different performance characteristics - caches provide fast access, while hard disks are more suitable for long-term storage of large amounts of infrequently accessed data. To illustrate this process more specifically, we can imagine a practical application scenario: in an enterprise-level server, multi-layer stacked memory is used as the main data storage medium. When a user uploads a new file or new data is generated within the system, this data becomes the target transmission data. At this time, by analyzing the previous access records of similar types of data (i.e., historical access characteristic parameters), the system can predict which new data may be used frequently and which may be idle for a long time. As a result, data that is expected to be used frequently will be placed on a faster access level, such as SSD or DRAM, while data that is expected to be less accessed will be arranged on a more economical storage level such as HDD. In this way, not only the response speed of the overall system is improved, but also the effective use of resources is achieved.

[0066] Step S2, using a bidirectional hash mapping algorithm, performs storage location mapping on the target transmission data based on the access frequency clustering result to obtain a multi-layer storage allocation strategy table.

[0067] Specifically, when implementing the steps described above, we continue to explore how to map the storage location of the target transmission data based on the access frequency clustering results through the bidirectional hash mapping algorithm, and finally obtain a multi-layer storage allocation strategy table. This process is based on the frequency clustering analysis that has been completed before, that is, we already have the classification results of which data belongs to hot data (frequently accessed data) and which data belongs to cold data (less frequently accessed data). Next, in order to determine which level of storage medium these data with different access frequencies should be placed on, we will use the bidirectional hash mapping algorithm. The role of this algorithm is that it can find the storage location that best suits their characteristics for each type of data based on the access frequency clustering results. Specifically, bidirectional hash mapping is a mapping method that can operate efficiently in both directions. It can map the input target transmission data to one of multiple predefined outputs, which here points to different levels in the multi-layer stacked memory. This mapping is not random, but is based on the data access pattern revealed by the previous clustering analysis. For example, for data blocks identified as hot data, the algorithm tends to map them to faster but potentially more expensive storage tiers, such as DRAM or SSD; while for cold data, more economical and larger HDD or other persistent storage tiers will be selected. When the bidirectional hash mapping algorithm completes the mapping of all target transfer data, the system will generate a multi-layer storage allocation strategy table. This table is actually a detailed guide that records the specific location where each piece of data should be stored - that is, in which tier of storage. This strategy table not only takes into account the access frequency of data, but also comprehensively considers factors such as the speed, capacity and cost of each storage tier to ensure the overall optimal performance. For example, in an enterprise-level server environment, when we process a large number of new files uploaded by users or new data generated within the system, with the help of the multi-layer storage allocation strategy table, we can accurately determine which data should be stored in the fast access tier first to respond to user requests in a timely manner, and which data can be safely stored in the slower but more cost-effective tier.

[0068] Step S3, performing interactive path planning for the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning.

[0069] Specifically, in the data interaction method of multi-layer stacked storage, once we generate a multi-layer storage allocation strategy table based on the bidirectional hash mapping algorithm, the next step is to plan the interaction path for the target transmission data based on this strategy table to obtain the optimal interaction path planning. This process is crucial because it determines the specific route of data movement between different levels, which directly affects the speed and efficiency of data access. To achieve this, the system carefully analyzes the information in the multi-layer storage allocation strategy table, which contains detailed instructions on which level each type of data should be stored. By interpreting these instructions, the system can build a path map that can guide data migration. In this process, the system not only takes into account the characteristics of the data itself, such as access frequency and size, but also combines factors such as the physical connection method between each storage level, transmission speed differences, and possible bottlenecks. For example, in an enterprise-level server environment, when a user uploads a new file or new data is generated within the system, the system will determine the best storage location for this data based on the multi-layer storage allocation strategy table. If the newly uploaded file is predicted to be hot data (frequently accessed data), the system will plan a path directly to the cache or SSD to ensure that this type of data can respond quickly to user access requests. At the same time, for cold data that is expected to be accessed less frequently, the system will choose a path to the HDD or other persistent storage layer. It is worth noting that optimal interactive path planning is not just a simple path selection, it also involves how to avoid potential conflicts and congestion problems. For example, if multiple data blocks need to be migrated to the same storage layer at the same time, the system will adjust their migration order or find alternative paths to ensure the efficient operation of the entire system. In addition, in order to further optimize performance, the system may use pre-fetching technology and intelligent scheduling algorithms to prepare the resources required for data migration in advance, and dynamically adjust the path planning to adapt to real-time changing workload conditions. In this way, whether in high concurrent access or low traffic periods, the system can maintain stable performance and ensure that users get a fast and consistent service experience. In this way, we can see that the interactive path planning based on the multi-layer storage allocation strategy table does achieve the optimization of the data interaction path.

[0070] Step S4: performing hierarchical migration scheduling on the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue.

[0071] Specifically, in the data interaction method of the multi-layer stacked memory, the process of hierarchical migration scheduling of the target transmission data based on the optimal interaction path planning is a key link in the entire data management strategy. This process is immediately after the optimal interaction path is determined. The core is to organize and arrange the migration activities of data from one storage level to another storage level according to the pre-planned path, and finally form an orderly data migration execution queue. In order to achieve this, the system needs to comprehensively consider multiple factors, including but not limited to the importance of data, access frequency, and physical connection characteristics between various storage levels. Specifically, when the system has obtained the optimal interaction path planning, it will start to analyze all target transmission data to be migrated, and formulate a detailed migration plan based on the movement requirements of these data between different levels. For example, in an enterprise-level server environment, assuming that a batch of newly uploaded files are identified as hot data (frequently accessed data), the system will prioritize the migration of this batch of data to the fast-responding SSD or DRAM level based on the optimal interaction path planning. At the same time, for those predicted to be cold data (less frequently accessed data), their migration to HDD or other persistent storage layers will be planned. In this process, the system not only has to decide which data should be migrated first, but also has to consider how to avoid multiple data blocks competing for the same resources at the same time, causing bottlenecks or delays. In order to ensure the efficiency and stability of the migration process, the system will build a data migration execution queue, which is like a command center responsible for coordinating all data migration tasks. Each entry in the queue represents a specific migration operation, including information such as the data identifier to be migrated, the source storage location, the target storage location, and the expected completion time. In this way, even in the face of a large number of concurrent data migration requests, the system can handle each task in an orderly manner to ensure that the overall performance is not affected. In addition, the system may also introduce a dynamic adjustment mechanism that allows tasks in the queue to be reordered according to the actual workload at runtime to adapt to changing needs. For example, if a large number of users suddenly request to access a certain part of the data, the system can immediately adjust the migration priority of this part of the data and migrate it to a storage layer closer to the user in advance, thereby speeding up the response. In summary, by scheduling the hierarchical migration of the target transmission data and forming a data migration execution queue, the system not only realizes the efficient transfer of data between different storage levels, but also ensures that each migration follows the most optimized path and maximizes the use of storage resources.

[0072] Step S5: performing parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism.

[0073] Specifically, in the data interaction method of the multi-layer stacked memory, parallel data transmission of the data migration execution queue through the asynchronous pipe transmission mechanism is an important step to ensure the efficient operation of the entire system. When the data migration execution queue has been constructed, it contains a series of migration tasks to be processed, and each task clearly specifies the data identifier to be migrated, the source storage location, the target storage location, and the expected completion time. At this time, in order to speed up the transfer process of data from one storage level to another and make full use of the bandwidth resources of the storage system, the system will use the asynchronous pipe transmission mechanism to implement parallel data transmission. Specifically, the asynchronous pipe transmission mechanism allows multiple data blocks to be transmitted on different paths at the same time without waiting for the previous data block to complete the transmission before starting the next one. This non-blocking data processing method greatly improves the efficiency of data transmission because it reduces the overall delay caused by waiting for a single long and time-consuming operation. For example, in an enterprise-level server environment, when a batch of hot data (frequently accessed data) needs to be migrated to a fast-responding SSD or DRAM level, the system can use the asynchronous pipe transmission mechanism to start multiple migration tasks at the same time. These tasks may involve different data blocks, but they can all be transmitted independently along their own optimal interactive path planning, thus avoiding the bottleneck problem that may be caused by traditional sequential transmission. In addition, the asynchronous pipe transmission mechanism also has good fault tolerance and flexibility. If a path has a temporary failure or performance degradation during the transmission process, the system can quickly identify this situation and dynamically adjust the task allocation on other paths to maintain the overall transmission rate. For example, if a path connected to the SSD becomes slow due to overload, the system can immediately redirect part of the migration tasks to another relatively idle path for continued transmission. Such flexibility not only ensures that all scheduled data migration tasks can be completed smoothly even in a complex and changeable network environment, but also optimizes the transmission strategy in real time according to the actual workload, further improving the system's response speed and user experience. To better understand this process, we can imagine a scenario: in a large data center, a user uploads a large number of files, and the system decides to migrate some of the data predicted to be hot data to the cache layer after analysis. At this time, through the asynchronous pipe transmission mechanism, the system can handle multiple migration requests at the same time without affecting existing services. Each data block is efficiently transferred to the target storage tier along the pre-planned optimal path, achieving rapid deployment and immediate availability. At the same time, the system continuously monitors all ongoing transfer activities. Once any abnormal conditions are detected, such as network congestion or hardware failure, it will immediately take measures to adjust to ensure the security and stability of the data migration process.

[0074] In a specific embodiment, performing frequency cluster analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results includes:

[0075] Dividing the historical access characteristic parameters into time windows to obtain a dynamic time window sequence;

[0076] Performing data access frequency analysis on the dynamic time window sequence by using a maximum entropy clustering algorithm to obtain a data access frequency distribution diagram;

[0077] Based on the data access frequency distribution diagram, the historical access characteristic parameters are divided into intervals to obtain a frequency interval division result; wherein the frequency interval division result includes a high-frequency access interval, a medium-frequency access interval and a low-frequency access interval;

[0078] Based on the frequency interval division result, frequency clustering analysis is performed on the target transmission data to obtain access frequency clustering results; wherein the access frequency clustering results include hot data, warm data and cold data.

[0079] Specifically, in the data interaction method of the multi-layer stacked memory, the process of performing frequency clustering analysis on the target transmission data based on the historical access characteristic parameters is a key step to achieve efficient data management and optimize storage performance. This process first involves time window segmentation of the historical access characteristic parameters to obtain a dynamic time window sequence. This means that the system will divide the access records within a period of time (such as one day, one week or one month) into multiple sub-intervals according to a fixed time interval, and each sub-interval is a dynamic time window. In this way, the changes in access patterns within different time periods can be captured, thereby more accurately reflecting the actual use of data. Next, in order to further analyze the information in these dynamic time window sequences, the system will use the maximum entropy clustering algorithm to perform data access frequency analysis, and then generate a data access frequency distribution diagram. The maximum entropy clustering algorithm is a statistical learning method that can effectively process data sets with high uncertainty and complexity. In this process, the algorithm will calculate the access frequency of each data block based on the number of accesses and access patterns in each time window, and display these frequencies in a graphical manner. For example, in an enterprise server environment, if a file is frequently read in the past week, its access frequency in the corresponding dynamic time window sequence will be high, and this will also be shown in the data access frequency distribution graph. With the data access frequency distribution graph, the system will divide the historical access characteristic parameters into intervals based on this graph to obtain the frequency interval division results. This division process aims to identify which data belongs to the high-frequency access interval, the medium-frequency access interval, and the low-frequency access interval. The high-frequency access interval usually corresponds to data that is requested almost every time, the data in the medium-frequency access interval has a relatively stable access volume, and the low-frequency access interval is data that is rarely touched. This division is crucial for the subsequent frequency clustering analysis because it provides a clear standard to distinguish different types of data. Finally, based on the frequency interval division results, the system will perform frequency clustering analysis on the target transmission data and finally obtain the access frequency clustering results. This step maps the previously determined high-frequency, medium-frequency, and low-frequency access intervals to specific data classifications - hot data, warm data, and cold data. Hot data refers to data that is frequently accessed and requires a quick response; warm data also has a certain access frequency, but not as high as hot data; cold data refers to data that has not been accessed for a long time. For example, in the application scenario of a data center, when a user uploads a batch of new files, the system will perform frequency clustering analysis based on their historical access patterns of similar files in the past. Files that are expected to be frequently used will be classified as hot data and will be stored preferentially in fast storage layers such as cache or SSD; while files with less frequent access will be arranged in HDD or other persistent storage layers. In this way, not only the response speed of the system is improved, but also the effective use of resources is achieved.

[0080] In a specific embodiment, the target transmission data is mapped to a storage location based on the access frequency clustering result by a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table, including:

[0081] Performing location-sensitive hash coding on the access frequency clustering results to obtain an initial hash mapping table, and performing consistent hash ring construction on the initial hash mapping table to obtain a ring address space; wherein the ring address space includes virtual node distribution identifiers, physical address mapping relationships, and conflict area markers;

[0082] Performing conflict detection on the annular address space through a Bloom filter to obtain a conflict detection result, and constructing a skip table index on the conflict detection result to obtain a multi-level index structure; wherein the multi-level index structure includes a hot data index layer, a warm data index layer, and a cold data index layer;

[0083] If a hash bucket overflow occurs in the load factor of any layer of the multi-level index structure, the multi-level index structure is dynamically bucket-split by an extensible hash algorithm to obtain a hash bucket sequence, and a load balancing analysis is performed on the hash bucket sequence to obtain balancing factor data; wherein the balancing factor data includes storage capacity distribution, access load distribution and bandwidth utilization;

[0084] The balancing factor data is collaboratively optimized to obtain an optimized weight vector, and a storage location mapping is performed on the target transmission data based on the optimized weight vector through a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table.

[0085] Specifically, in the data interaction method of the multi-layer stacked memory, the process of mapping the storage location of the target transmission data based on the access frequency clustering result through the bidirectional hash mapping algorithm is the core link to achieve efficient data management and optimize storage performance. In order to explain this process in detail, we need to start with the locality-sensitive hash coding of the access frequency clustering result. First, the system will perform locality-sensitive hashing (LSH) encoding on the access frequency clustering result to obtain an initial hash mapping table. Locality-sensitive hashing is a technology for quickly finding similar items, which can reduce the computational complexity while maintaining data similarity. At this stage, the system converts hot data, warm data, and cold data into a series of hash values ​​according to their access patterns, and these hash values ​​can reflect the similarity and distribution characteristics between the data. For example, in an enterprise-level server environment, when a batch of files are classified into data with different access frequencies, the locality-sensitive hash coding will generate a corresponding hash representation for each type of file, thereby providing a basis for subsequent mapping. Subsequently, the system will construct a consistent hash ring based on this initial hash mapping table to obtain a ring address space containing virtual node distribution identifiers, physical address mapping relationships, and conflict area markers. The consistent hash ring is an effective mechanism to solve the data allocation problem in distributed systems. It evenly distributes data by introducing virtual nodes and ensures that the amount of data migration can be minimized even when nodes are increased or decreased. In this ring structure, each hash value is mapped to a position on the ring, and the actual data is allocated to the nearest physical or virtual node according to the interval of its hash value. This not only improves the fault tolerance of the system, but also allows data to be flexibly migrated between multiple storage levels. For example, when a batch of newly uploaded files are hashed, the system determines their specific locations in the ring address space based on their hash values ​​and allocates them to the most suitable storage level. Next, in order to deal with possible hash collision problems, the system performs conflict detection on the ring address space through a Bloom filter to obtain a conflict detection result. A Bloom filter is a probabilistic data structure that can efficiently determine whether an element exists in a set while occupying very little space. Here, the Bloom filter is used to check whether there is duplication of hash values, that is, whether different data is mapped to the same physical address. Once a conflict is detected, the system starts the skip list index building process, resulting in a multi-level index structure that includes hot data index layer, warm data index layer, and cold data index layer. A skip list is an improved version of a linked list that allows fast search, insertion, and deletion operations, making it ideal for managing large-scale data sets. In our application scenario, this multi-level index structure not only helps to improve data retrieval speed, but also further refines the management strategy for data with different access frequencies.For example, for hot data that needs to be accessed frequently, the system can establish a more sophisticated index layer to ensure fast positioning; while for cold data that is accessed less frequently, a looser index rule can be used to save resources. However, as the amount of data grows, any layer of the multi-level index structure may overflow the hash bucket, that is, the data stored in a single hash bucket exceeds the preset capacity limit. At this time, the system will dynamically split the multi-level index structure through the scalable hash algorithm to obtain a hash bucket sequence, and perform load balancing analysis on this sequence to obtain the balancing factor data. The scalable hash algorithm allows the hash table to add new hash buckets without destroying the existing mapping relationship, thereby effectively coping with the challenges brought by data growth. The balancing factor data analyzes key indicators such as storage capacity distribution, access load distribution, and bandwidth utilization, aiming to ensure that the resources of the entire storage system are reasonably utilized. For example, if the number of hash buckets in a certain layer of index is too large, resulting in increased access latency, the system can optimize performance by adjusting the number and distribution of hash buckets to ensure that each storage level can operate stably. Finally, based on the above balancing factor data, the system will perform collaborative optimization calculations on these data to obtain an optimized weight vector. This weight vector takes into account a variety of factors, such as data access frequency, storage cost, response time, etc., to guide the final storage location mapping decision. Through the bidirectional hash mapping algorithm, the system will map the storage location of the target transmission data according to the optimized weight vector to obtain a multi-layer storage allocation strategy table. This mapping process not only ensures that the data can be placed in the most appropriate location according to its access characteristics and system requirements, but also realizes the optimal configuration of multi-layer storage resources. For example, in the application scenario of the data center, when a user uploads a batch of files, the system will accurately determine which files should be stored in the cache layer first and which can be placed in the slower but more cost-effective storage layer based on their access frequency and other attributes combined with the optimized weight vector. In this way, not only the response speed of the system is improved, but also the best utilization of resources is achieved.

[0086] In a specific embodiment, performing interactive path planning on the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning includes:

[0087] Based on the multi-layer storage allocation strategy table, an initial path is constructed for the target transmission data to obtain a set of candidate paths, and a path delay prediction is performed on the set of candidate paths to obtain a path delay prediction table; wherein the path delay prediction table includes an estimated transmission delay, an estimated queuing delay, and an estimated congestion delay of each candidate path;

[0088] Performing path search and optimization on the path delay prediction table by using an ant colony optimization algorithm to obtain a path efficiency ranking list, and performing energy consumption balance analysis on the path efficiency ranking list to obtain an energy consumption balance evaluation table; wherein the energy consumption balance evaluation table includes the energy consumption level of each candidate path and the energy consumption distribution of the storage level;

[0089] Pruning the path nodes in the path efficiency ranking list based on the energy consumption balance evaluation table to obtain a pruned path set;

[0090] Performing path topology optimization on the pruned path set by a genetic algorithm to obtain an optimized path topology structure, and performing bandwidth resource allocation on the optimized path topology structure to obtain a bandwidth resource allocation table; wherein the bandwidth resource allocation table includes bandwidth allocation conditions and bandwidth utilization efficiency of each path segment;

[0091] Based on the bandwidth resource allocation table, QoS policy configuration is performed on the optimized path topology structure to obtain a QoS policy configuration table, and path stability evaluation is performed on the QoS policy configuration table to obtain a path stability report; wherein the QoS policy configuration table includes a service quality level, priority setting, and fault tolerance mechanism for each path segment;

[0092] The optimal path is selected for the path stability report through a dynamic programming algorithm to obtain an optimal interactive path planning; wherein the optimal interactive path planning includes a selected optimal path and a backup path.

[0093] Specifically, in the data interaction method of the multi-layer stacked memory, the process of interactive path planning for the target transmission data based on the multi-layer storage allocation strategy table is a key step to ensure efficient and stable data migration. This process not only involves selecting the optimal path from many possible paths, but also must consider multiple factors such as path delay, energy consumption balance, bandwidth resource allocation, and quality of service (QoS) to achieve the best data transmission performance. In order to explain this complex process in detail, we will start with the initial path construction and gradually explore the specific implementation methods of each stage. First, the system will perform an initial path construction for the target transmission data based on the multi-layer storage allocation strategy table to obtain a set of candidate paths. This means that for each data block that needs to be migrated, the system will calculate several possible migration paths according to the pre-established storage level distribution strategy. Each candidate path represents a specific route from the source storage location to the target storage location. Subsequently, in order to evaluate the actual transmission efficiency of these paths, the system will perform path delay prediction on the candidate path set and generate a path delay prediction table. This table records the estimated transmission delay, estimated queuing delay, and estimated congestion delay of each candidate path, providing an important reference for subsequent optimization. For example, in an enterprise-level server environment, when a batch of files is identified as hot data, the system will build multiple candidate paths to the fast response level (such as SSD or DRAM) for them, and estimate the possible time delay on each path through simulation calculation, so as to preliminarily screen out potential efficient paths. Next, in order to further optimize the path selection, the system will use the ant colony optimization algorithm to search and optimize the path delay prediction table to obtain a path efficiency ranking list. The ant colony optimization algorithm is a heuristic search method that simulates the foraging behavior of ants in nature. It can effectively handle path optimization problems in complex environments. In this process, the algorithm will simulate multiple "ants" exploring along different paths, dynamically adjust the path selection probability according to the data in the path delay prediction table, and finally find the path combination with the highest efficiency. In order to ensure the energy efficiency balance of the entire system, the system will also perform energy consumption balance analysis on the path efficiency ranking list to obtain an energy consumption balance evaluation table. The evaluation table covers the energy consumption level of each candidate path and the energy consumption distribution of the storage layer, helping the system understand which paths may cause excessive energy consumption, thereby avoiding unnecessary resource waste. For example, in a data center application scenario, if a path has a fast transmission speed but high energy consumption, the system can adjust the path selection to give priority to those paths that can ensure transmission efficiency and maintain low energy consumption, so as to optimize the overall energy consumption. With the energy consumption balance evaluation table, the system will prune the path nodes based on the path efficiency sorting list to obtain the pruned path set. Path node pruning refers to removing those path nodes that obviously do not meet the energy consumption requirements or cannot provide sufficient performance gains, reducing unnecessary calculations and resource usage.This step not only simplifies the subsequent optimization process, but also improves the accuracy of the final path selection. Next, the system will use genetic algorithms to optimize the path topology of the pruned path set to obtain the optimized path topology. Genetic algorithms are a global search method that imitates the biological evolution process. It continuously iterates to produce better solutions by performing operations such as crossover and mutation on the elements in the path set. In this process, the system will also allocate bandwidth resources to the optimized path topology and generate a bandwidth resource allocation table. This table includes the bandwidth allocation and bandwidth utilization efficiency of each path segment, ensuring that each part of the data transmission can obtain sufficient bandwidth support while maximizing the use of existing network resources. For example, in the case of high concurrent access, the system can flexibly adjust bandwidth allocation according to real-time traffic conditions to prioritize the transmission speed of important data. Finally, based on the bandwidth resource allocation table, the system will configure QoS policies for the optimized path topology and generate a QoS policy configuration table. QoS policy configuration aims to set appropriate service quality levels, priority settings, and fault tolerance mechanisms for different data transmission tasks to ensure stable transmission performance even when network conditions change. In order to verify the effectiveness of these configurations, the system will evaluate the path stability of the QoS policy configuration table and generate a path stability report. This report describes in detail the performance of each path segment under different load conditions, providing a reliable basis for the final path selection. By using a dynamic programming algorithm to select the optimal path for the path stability report, the system can determine the optimal interactive path planning, including the selected optimal path and backup paths. For example, in an enterprise-level server environment, when a user uploads a new file, the system will not only select the shortest and fastest primary path for the initial transmission, but also prepare one or more backup paths to deal with possible failures or bottlenecks, ensuring that data can be safely and quickly migrated to the specified storage tier while maintaining the efficient operation of the entire system.

[0094] In a specific embodiment, the hierarchical migration scheduling of the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue includes:

[0095] Based on the optimal interactive path planning, the target transmission data is divided into data blocks to obtain a data block set, and the data block set is prioritized to obtain a priority queue;

[0096] Allocating time slices to the priority queues by a time slice round-robin algorithm to obtain a time slice allocation table, and performing resource competition prediction on the time slice allocation table to obtain a resource competition prediction graph; wherein the resource competition prediction graph includes resource requirements, potential resource conflicts, and competition levels of each of the data block sets;

[0097] The scheduling strategy of the time slice allocation table is adjusted based on the resource competition prediction graph to obtain an adjusted time slice allocation table, and data migration dependency analysis is performed on the adjusted time slice allocation table to obtain a data migration dependency graph; wherein the data migration dependency graph is used to describe the dependency and sequence between the data block sets during the migration process;

[0098] Sorting the data migration dependency graph in migration order by using a topological sorting algorithm to obtain a data migration order list;

[0099] The data migration order list is encapsulated as a data migration task to obtain a data migration task queue, and the data migration task queue is hierarchically scheduled through a multi-level feedback queue scheduling algorithm to obtain a data migration execution queue.

[0100] Specifically, in the data interaction method of the multi-layer stacked memory, the process of hierarchical migration scheduling of the target transmission data based on the optimal interaction path planning is the core link to ensure efficient and orderly data migration. This process not only requires accurate management of the migration order and priority of each data block, but also requires consideration of resource competition, dependencies, and overall load balancing of the system. In order to explain this complex process in detail, we will start with data segmentation and gradually explore the specific implementation methods of each stage. First, the system will segment the target transmission data based on the optimal interaction path planning to obtain a data block set. This means that for each data file or object that needs to be migrated, the system will divide it into multiple smaller data blocks for independent processing and parallel transmission. Subsequently, the system will prioritize these data block sets and generate a priority queue. Prioritization is based on the importance of each data block, access frequency, and other factors that may affect migration efficiency. For example, in an enterprise-level server environment, when a batch of newly uploaded files are identified as hot data, the system will divide the files into multiple small blocks according to their expected access frequency, and give higher priority to those data blocks that are expected to be frequently accessed, ensuring that they can be migrated to the fast response level (such as SSD or DRAM) faster to meet the immediate needs of users. Next, in order to effectively manage and schedule these data blocks with different priorities, the system will allocate time slices to the priority queues through the time slice round-robin algorithm to obtain a time slice allocation table. Time slice round-robin is a common CPU scheduling algorithm that allows each task to run within a certain time slice and then switch to the next task. In this process, the system will allocate the corresponding time slice length to each data block according to its priority, so as to ensure that high-priority data blocks can get more processing time. At the same time, the system will also predict resource competition for the time slice allocation table and generate a resource competition prediction graph. This graph includes information such as resource requirements, potential resource conflicts, and the degree of competition for each data block set, helping the system to identify possible bottleneck problems in advance. For example, in the application scenario of the data center, if multiple high-priority data blocks request the same storage resources at the same time, the system can adjust their transmission order in advance through the resource competition prediction graph to avoid delays or failures caused by resource contention. With the resource competition prediction graph, the system will adjust the scheduling strategy of the time slice allocation table based on it to obtain the adjusted time slice allocation table. This step aims to optimize resource utilization, reduce conflicts, and improve overall migration efficiency. Subsequently, the system will perform data migration dependency analysis on the adjusted time slice allocation table and generate a data migration dependency graph. This graph is used to describe the dependency and sequence between the various data block sets during the migration process, ensuring that the associated data blocks can be migrated in the correct order. For example, some data blocks may be prerequisites for other data blocks, and the migration must be completed before the subsequent operations can continue.By constructing this dependency graph, the system can ensure that all dependencies are properly handled and prevent migration interruptions caused by unsatisfied dependencies. In order to further optimize the migration order, the system will sort the migration order of the data migration dependency graph through a topological sorting algorithm to obtain a data migration order list. Topological sorting is a sorting method applicable to directed acyclic graphs. It can determine a linear sequence based on the dependencies between nodes so that all predecessor nodes of each node appear before it. In this process, the system will calculate a reasonable migration order based on the information in the dependency graph to ensure that all data blocks can be successfully migrated as needed. For example, in an enterprise-level server environment, when there are multiple interrelated data blocks, the system will determine their optimal migration order through a topological sorting algorithm to avoid unnecessary waiting time and improve the overall migration speed. Finally, the system will encapsulate the data migration task list into a data migration task queue, and perform hierarchical migration scheduling on the data migration task queue through a multi-level feedback queue scheduling algorithm to obtain a data migration execution queue. The multi-level feedback queue scheduling algorithm is a scheduling method that can dynamically adjust task priorities. It allows tasks to be flexibly moved between queues of different priorities to adapt to real-time changing workload conditions. During this process, the system will continuously monitor the status of each data migration task, dynamically adjust its queue according to the actual execution situation, and ensure that high-priority tasks can be completed as soon as possible, and low-priority tasks will not be postponed indefinitely. For example, in the application scenario of the data center, when a user uploads a new file, the system will not only select the shortest and fastest primary path for the initial transmission, but also prepare one or more backup paths to deal with possible failures or bottlenecks, ensuring that the data can be safely and quickly migrated to the specified storage level while maintaining the efficient operation of the entire system. In this way, not only the response speed of the system is improved, but also the best use of resources is achieved.

[0101] In a specific embodiment, the performing parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism includes:

[0102] Dividing the data migration execution queue into pipeline segments to obtain a pipeline segment sequence, and calculating a flow control threshold value for the pipeline segment sequence to obtain a flow control parameter set;

[0103] Performing traffic shaping processing on the flow control parameter set through a preset token bucket algorithm to obtain a transmission rate control table, and dynamically allocating a buffer zone on the transmission rate control table to obtain a buffer zone configuration scheme;

[0104] Based on the buffer configuration scheme, a parallel transmission channel is constructed for the pipeline segment sequence to obtain a parallel transmission topology map, and a semaphore synchronization analysis is performed on the parallel transmission topology map to obtain a synchronization control strategy; wherein the synchronization control strategy includes mutually exclusive access control, synchronization signal triggering conditions and deadlock avoidance mechanism;

[0105] The synchronous control strategy is optimized for data transmission by using zero-copy technology to obtain an optimized transmission scheme, and the optimized transmission scheme is scheduled for transmission timing to obtain a pipeline transmission scheduling table;

[0106] Through a preset asynchronous pipeline transmission mechanism, parallel data transmission is performed on the data migration execution queue based on the pipeline transmission scheduling table.

[0107] Specifically, in the data interaction method of the multi-layer stacked memory, the process of parallel data transmission of the data migration execution queue through the asynchronous pipeline transmission mechanism is a key step to ensure efficient, stable and fast completion of large-scale data migration. In order to explain this complex process in detail, we will start with pipeline segmentation and gradually explore the specific implementation methods of each stage. First, the system will perform pipeline segmentation on the data migration execution queue to obtain a pipeline segment sequence. This means that the entire data migration task is decomposed into multiple smaller, independently processable fragments, each fragment representing a specific data block or a group of related data blocks. This segmentation strategy not only simplifies the management complexity, but also provides a basis for subsequent parallel processing. Subsequently, the system will calculate the flow control threshold for these pipeline segment sequences to obtain a flow control parameter set. The flow control threshold calculation is to ensure that the transmission between the segments will not cause bottlenecks or congestion due to overload. For example, in an enterprise-level server environment, when a batch of files need to be migrated from a low-speed storage layer to a high-speed cache layer, the system will set a reasonable flow control threshold based on factors such as network bandwidth and storage device performance to prevent any segment from occupying too many resources and affecting the transmission efficiency of other segments. Next, in order to further optimize the transmission rate, the system will perform traffic shaping on the flow control parameter set through the preset token bucket algorithm to obtain the transmission rate control table. The token bucket algorithm is a commonly used flow control mechanism that limits the speed of data transmission by issuing "tokens" to achieve smooth data flow transmission. In this process, the system will dynamically adjust the token issuance frequency and quantity according to the information in the flow control parameter set to ensure that each pipeline segment can be transmitted at the predetermined rate. At the same time, in order to cope with possible burst traffic, the system will also dynamically allocate buffers to the transmission rate control table to obtain a buffer configuration scheme. The buffer configuration scheme is designed to provide sufficient temporary storage space for each pipeline segment so that it can absorb excess load when traffic fluctuates and avoid transmission interruption or delay caused by instantaneous traffic peaks. For example, in the application scenario of a data center, if the transmission rate of a segment suddenly increases, the system can relieve the pressure by increasing the buffer capacity of the segment to ensure the stability of the overall transmission process. With the buffer configuration scheme, the system will build parallel transmission channels for the pipeline segment sequence based on this to obtain a parallel transmission topology diagram. Parallel transmission channel construction means creating independent transmission paths for each pipeline segment, so that they can be transmitted simultaneously at different time points or through different physical paths. This not only improves transmission efficiency, but also makes full use of network and storage resources. Subsequently, the system will perform semaphore synchronization analysis on the parallel transmission topology diagram to obtain the synchronization control strategy. The synchronization control strategy includes elements such as mutual exclusive access control, synchronization signal triggering conditions, and deadlock avoidance mechanism to ensure that no conflicts or blocking problems occur during parallel transmission.For example, in a multi-threaded or distributed environment, different pipeline segments may compete for the same resources. The system can ensure that only one segment can access the resource at a time by setting up mutually exclusive access control. The synchronization signal trigger condition is used to coordinate the operation order between the segments to ensure that they can complete the transmission task in an orderly manner. The deadlock avoidance mechanism is to prevent possible circular waiting situations and ensure the normal operation of the entire system. In order to further improve the transmission efficiency, the system will optimize the data transmission of the synchronization control strategy through zero-copy technology to obtain an optimized transmission plan. Zero-copy technology is a method to reduce the number of data copies. It allows data to be transmitted directly from the source address to the target address without going through an intermediate cache or additional memory copy operation. This not only saves CPU resources, but also significantly reduces transmission delays. On this basis, the system will schedule the transmission timing of the optimized transmission plan to obtain a pipeline transmission scheduling table. This table records the transmission order, time, priority and other information of each pipeline segment in detail to ensure that all segments can complete the transmission according to the optimal path and timing. For example, in an enterprise server environment, when a user uploads a new file, the system will not only select the shortest and fastest primary path for the initial transfer, but also prepare one or more backup paths to deal with possible failures or bottlenecks, ensuring that data can be safely and quickly migrated to the specified storage tier while maintaining the efficient operation of the entire system. Finally, the system will perform parallel data transmission on the data migration execution queue based on the pipeline transmission schedule through the preset asynchronous pipeline transmission mechanism. The asynchronous pipeline transmission mechanism allows each pipeline segment to be transmitted independently without waiting for the previous segment to complete before starting the next segment. This method greatly improves the concurrency and flexibility of data transmission, and can successfully complete all scheduled data migration tasks even in complex and changeable network environments. For example, in the application scenario of a data center, when a large number of files need to be migrated at the same time, the system can simultaneously start multiple transmission channels through the asynchronous pipeline transmission mechanism to ensure that each file can reach its target location as soon as possible, thereby improving the overall migration speed and user experience. In this way, not only the system response speed is improved, but also the best use of resources is achieved.

[0108] In a specific embodiment, the flow control parameter set is subjected to traffic shaping processing by a preset token bucket algorithm to obtain a transmission rate control table, including:

[0109] Calculating the token generation rate for the flow control parameter set to obtain a token generation periodic table;

[0110] The bucket capacity of the token generation periodic table is calculated by a dynamic bucket depth adjustment algorithm to obtain a token bucket depth distribution diagram;

[0111] Performing burst traffic processing on the token bucket depth distribution graph to obtain a burst traffic control strategy, which includes a maximum burst capacity, a burst traffic buffer size, and a traffic smoothing factor;

[0112] The burst flow control strategy is allocated tokens by a preset token bucket algorithm to obtain token allocation weight data; wherein the token allocation weight data includes priority weight, delay sensitivity and throughput requirement;

[0113] Performing traffic shaping processing on the target transmission data based on the token allocation weight data to obtain a traffic shaping parameter set;

[0114] The traffic shaping parameter set is rate constrained by a layered rate limiting algorithm to obtain a transmission rate control table; wherein the transmission rate control table includes rate limits for each layer, rate ratios between layers, and rate adjustment steps.

[0115] Specifically, when implementing the flow control mechanism, the flow control parameter set is shaped by the preset token bucket algorithm to ensure efficient and stable data transmission in network communication. This process starts from calculating the token generation rate of the flow control parameter set and ends with the final formation of the transmission rate control table. Each step is to optimize and manage the behavior of data flow in the network. When the system needs to perform traffic shaping on the flow control parameter set, it first calculates the token generation rate according to the content of the flow control parameter set to obtain the token generation periodic table. The token generation rate here refers to how many tokens the system can generate for a specific data flow per unit time, and these tokens represent the amount of data allowed to be transmitted. For example, in a video conferencing application, if the system is configured with a generation rate of 100 tokens per second, it means that the data flow of the application can consume 100 tokens per second to send data. The token generation periodic table records the specific rate of token generation under different conditions for subsequent processing. Next, the system uses the dynamic bucket depth adjustment algorithm to calculate the bucket capacity of the token generation periodic table to obtain a token bucket depth distribution diagram. The dynamic bucket depth adjustment algorithm is an intelligent algorithm that can dynamically adjust the capacity of the token bucket according to the network conditions and the current data traffic. The depth of the token bucket determines how many unused tokens can be accumulated at the same time, which affects the processing capacity of burst traffic. Continuing with the video conference as an example, if a large amount of data exchange needs to suddenly appear during the conference (such as screen sharing), the token bucket depth distribution diagram will show that the capacity of the token bucket increases at this time so that there are enough tokens to support this sudden situation. With the token bucket depth distribution diagram, the system will further process the burst traffic to obtain the burst traffic control strategy. This step involves setting parameters such as maximum burst capacity, burst traffic buffer size and traffic smoothing factor. The maximum burst capacity defines the maximum amount of data that can burst in a short period of time; the burst traffic buffer size refers to the space used by the system to temporarily store burst traffic; the traffic smoothing factor is used to control how to smooth the burst traffic so that it is more evenly distributed on the timeline. For example, if there is a high-resolution video clip that needs to be transmitted in a video conference, the system will use a larger burst capacity and an appropriate buffer to cope with this instantaneous peak according to the pre-set burst traffic control strategy, and ensure that the subsequent data flow can smoothly transition through the traffic smoothing factor. Then, the system will allocate tokens for the burst flow control strategy based on the preset token bucket algorithm to obtain token allocation weight data. The token allocation weight data contains information such as priority weight, delay sensitivity, and throughput requirements, which together determine which data streams can obtain more token resources. For example, in a video conferencing scenario, the voice stream may be given a higher priority weight and lower delay sensitivity because it requires real-time performance and low latency; while the video stream has a higher throughput requirement, but its delay sensitivity is relatively low.Therefore, when allocating tokens, the system will prioritize the smoothness of the voice stream while trying to meet the large data volume transmission requirements of the video stream as much as possible. Based on the above token allocation weight data, the system will perform traffic shaping processing on the target transmission data to obtain a traffic shaping parameter set. Traffic shaping processing aims to adjust the speed and mode of the data stream according to predetermined rules to meet the requirements of the network and avoid congestion. In this process, the system will refer to various factors in the token allocation weight data to formulate personalized traffic shaping plans for different data streams. For video conferencing, this means that the system will ensure that both audio and video streams can obtain appropriate bandwidth while maintaining the quality and stability of the entire conference. Finally, in order to further refine the control, the system will rate constrain the traffic shaping parameter set through a layered rate limiting algorithm to obtain a transmission rate control table. The layered rate limiting algorithm allows the system to implement differentiated rate management for data streams at different levels, including rate limits at each layer, rate ratios between layers, and rate adjustment steps. For example, in a video conferencing environment, the system may set different rate limits for basic control signaling, main audio and video streams, and auxiliary information streams to ensure that each part can operate in the best state. As the result of this series of operations, the transmission rate control table not only summarizes all rate-related decisions, but also provides clear guidelines for actual data transmission, ensuring that data flows in application scenarios such as video conferencing can be transmitted smoothly in complex and changing network environments. In this way, the system response speed is improved, and the best use of resources is achieved, demonstrating the value and effect of the technical solution in practical applications.

[0116] The above describes the data interaction method of the multi-layer stacked memory in the embodiment of the present invention. The following describes the data interaction device of the multi-layer stacked memory in the embodiment of the present invention. Figure 2 , an embodiment of a data interaction device of a multi-layer stacked memory in an embodiment of the present invention includes:

[0117] The acquisition module 21 is used to obtain the historical access characteristic parameters of the storage layer in the target memory and the target transmission data input into the target memory, and perform frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein the multi-layer stacked memory is used as the target memory;

[0118] A mapping module 22, configured to perform storage location mapping on the target transmission data based on the access frequency clustering result by using a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table;

[0119] A planning module 23, configured to perform interactive path planning for the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning;

[0120] A scheduling module 24, configured to perform hierarchical migration scheduling on the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue;

[0121] The transmission module 25 is used to perform parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism.

[0122] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0123] Reference Figure 3 The present invention also provides a computer device in an embodiment, wherein the internal structure of the computer device can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0124] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0125] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0127] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0128] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data interaction method for a multi-layer stacked memory, characterized in that: The following steps are involved: Obtaining historical access characteristic parameters of the storage level in the target memory and target transmission data input into the target memory, and performing frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein the multi-layer stacked memory is used as the target memory; By using a bidirectional hash mapping algorithm, the target transmission data is mapped to a storage location based on the access frequency clustering result to obtain a multi-layer storage allocation strategy table; Performing interactive path planning on the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning; Based on the multi-layer storage allocation strategy table, an initial path is constructed for the target transmission data to obtain a candidate path set, and a path delay prediction is performed on the candidate path set to obtain a path delay prediction table; Performing path search and optimization on the path delay prediction table by using an ant colony optimization algorithm to obtain a path efficiency ranking list, and performing energy consumption balance analysis on the path efficiency ranking list to obtain an energy consumption balance evaluation table; Pruning the path nodes in the path efficiency ranking list based on the energy consumption balance evaluation table to obtain a pruned path set; Performing path topology optimization on the pruned path set by a genetic algorithm to obtain an optimized path topology structure, and performing bandwidth resource allocation on the optimized path topology structure to obtain a bandwidth resource allocation table; Based on the bandwidth resource allocation table, QoS policy configuration is performed on the optimized path topology structure to obtain a QoS policy configuration table, and path stability evaluation is performed on the QoS policy configuration table to obtain a path stability report; wherein the QoS policy configuration table includes a service quality level, priority setting, and fault tolerance mechanism for each path segment; Performing optimal path selection on the path stability report through a dynamic programming algorithm to obtain an optimal interactive path planning; wherein the optimal interactive path planning includes a selected optimal path and a backup path; Based on the optimal interaction path planning, hierarchical migration scheduling is performed on the target transmission data to obtain a data migration execution queue; The data migration execution queue is subjected to parallel data transmission via an asynchronous pipeline transmission mechanism.

2. The data interaction method of the multi-layer stacked memory according to claim 1, characterized in that: The performing frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results includes: Dividing the historical access characteristic parameters into time windows to obtain a dynamic time window sequence; Performing data access frequency analysis on the dynamic time window sequence by using a maximum entropy clustering algorithm to obtain a data access frequency distribution diagram; Based on the data access frequency distribution diagram, the historical access characteristic parameters are divided into intervals to obtain a frequency interval division result; wherein the frequency interval division result includes a high-frequency access interval, a medium-frequency access interval and a low-frequency access interval; Based on the frequency interval division result, frequency clustering analysis is performed on the target transmission data to obtain access frequency clustering results; wherein the access frequency clustering results include hot data, warm data and cold data.

3. The data interaction method of the multi-layer stacked memory according to claim 1, characterized in that: The target transmission data is mapped to a storage location based on the access frequency clustering result by a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table, including: Performing location-sensitive hash coding on the access frequency clustering results to obtain an initial hash mapping table, and performing consistent hash ring construction on the initial hash mapping table to obtain a ring address space; wherein the ring address space includes virtual node distribution identifiers, physical address mapping relationships, and conflict area markers; Performing conflict detection on the annular address space through a Bloom filter to obtain a conflict detection result, and constructing a skip table index on the conflict detection result to obtain a multi-level index structure; wherein the multi-level index structure includes a hot data index layer, a warm data index layer, and a cold data index layer; If a hash bucket overflow occurs in the load factor of any layer of the multi-level index structure, the multi-level index structure is dynamically bucket-split by an extensible hash algorithm to obtain a hash bucket sequence, and a load balancing analysis is performed on the hash bucket sequence to obtain balancing factor data; wherein the balancing factor data includes storage capacity distribution, access load distribution and bandwidth utilization; The balancing factor data is collaboratively optimized to obtain an optimized weight vector, and a storage location mapping is performed on the target transmission data based on the optimized weight vector through a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table.

4. The data interaction method of the multi-layer stacked memory according to claim 1, characterized in that: The step of performing hierarchical migration scheduling on the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue includes: Based on the optimal interactive path planning, the target transmission data is divided into blocks to obtain a data block set, and the data block set is prioritized to obtain a priority queue; Allocating time slices to the priority queues by a time slice round-robin algorithm to obtain a time slice allocation table, and performing resource competition prediction on the time slice allocation table to obtain a resource competition prediction graph; wherein the resource competition prediction graph includes resource requirements, potential resource conflicts, and competition levels of each of the data block sets; The scheduling strategy of the time slice allocation table is adjusted based on the resource competition prediction graph to obtain an adjusted time slice allocation table, and data migration dependency analysis is performed on the adjusted time slice allocation table to obtain a data migration dependency graph; wherein the data migration dependency graph is used to describe the dependency and sequence between the data block sets during the migration process; Sorting the data migration dependency graph in migration order by using a topological sorting algorithm to obtain a data migration order list; The data migration order list is encapsulated as a data migration task to obtain a data migration task queue, and the data migration task queue is hierarchically scheduled through a multi-level feedback queue scheduling algorithm to obtain a data migration execution queue.

5. The data interaction method of the multi-layer stacked memory according to claim 1, characterized in that: The performing parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism includes: Dividing the data migration execution queue into pipeline segments to obtain a pipeline segment sequence, and calculating a flow control threshold value for the pipeline segment sequence to obtain a flow control parameter set; Performing traffic shaping processing on the flow control parameter set through a preset token bucket algorithm to obtain a transmission rate control table, and dynamically allocating a buffer zone on the transmission rate control table to obtain a buffer zone configuration scheme; Based on the buffer configuration scheme, a parallel transmission channel is constructed for the pipeline segment sequence to obtain a parallel transmission topology map, and a semaphore synchronization analysis is performed on the parallel transmission topology map to obtain a synchronization control strategy; wherein the synchronization control strategy includes mutually exclusive access control, synchronization signal triggering conditions and deadlock avoidance mechanism; The synchronous control strategy is optimized for data transmission by using zero-copy technology to obtain an optimized transmission scheme, and the optimized transmission scheme is scheduled for transmission timing to obtain a pipeline transmission scheduling table; Through a preset asynchronous pipeline transmission mechanism, parallel data transmission is performed on the data migration execution queue based on the pipeline transmission scheduling table.

6. The data interaction method of the multi-layer stacked memory according to claim 5, characterized in that: The flow control parameter set is subjected to flow shaping processing by a preset token bucket algorithm to obtain a transmission rate control table, including: Calculating the token generation rate for the flow control parameter set to obtain a token generation periodic table; The bucket capacity of the token generation periodic table is calculated by a dynamic bucket depth adjustment algorithm to obtain a token bucket depth distribution diagram; Performing burst traffic processing on the token bucket depth distribution graph to obtain a burst traffic control strategy, which includes a maximum burst capacity, a burst traffic buffer size, and a traffic smoothing factor; The burst flow control strategy is allocated tokens by a preset token bucket algorithm to obtain token allocation weight data; wherein the token allocation weight data includes priority weight, delay sensitivity and throughput requirement; Performing traffic shaping processing on the target transmission data based on the token allocation weight data to obtain a traffic shaping parameter set; The traffic shaping parameter set is rate constrained by a layered rate limiting algorithm to obtain a transmission rate control table; wherein the transmission rate control table includes rate limits for each layer, rate ratios between layers, and rate adjustment steps.

7. A data interaction device for a multi-layer stacked memory, characterized in that: A data interaction method for implementing a multi-layer stacked memory according to any one of claims 1 to 6, comprising: an acquisition module, used to acquire historical access characteristic parameters of storage layers in a target memory and target transmission data input into the target memory, and to perform frequency clustering analysis on the target transmission data based on the historical access characteristic parameters to obtain access frequency clustering results; wherein the multi-layer stacked memory is used as the target memory; A mapping module, used to perform storage location mapping on the target transmission data based on the access frequency clustering result by a bidirectional hash mapping algorithm to obtain a multi-layer storage allocation strategy table; A planning module, used to perform interactive path planning for the target transmission data based on the multi-layer storage allocation strategy table to obtain an optimal interactive path planning; A scheduling module, configured to perform hierarchical migration scheduling on the target transmission data based on the optimal interaction path planning to obtain a data migration execution queue; The transmission module is used to perform parallel data transmission on the data migration execution queue through an asynchronous pipeline transmission mechanism.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images