Computer memory data processing system

Abstract

The invention relates to the technical field of computer storage, and discloses a computer memory data processing system, which comprises the following modules: an intelligent cache prefetching module, which is used for predicting a future memory access address based on deep learning and reinforcement learning technologies and loading data into a cache in advance; the heterogeneous storage management module is used for dynamically classifying the data based on the data popularity value and realizing migration of the data among different storage media; and the storage consistency guarantee module is used for managing the data consistency among different storage layers through the improved MES I protocol. Cache prefetching is optimized by introducing deep learning and reinforcement learning technologies, and by predicting a future memory access address in real time and loading data in advance, the system effectively reduces the cache missing rate, improves the cache hit rate, remarkably reduces the memory access delay, and improves the memory access efficiency. And the system can keep more efficient data access performance in a large-scale data processing scene.

Description

Technical Field

[0001] The present invention relates to the technical field of computer storage, and in particular to a computer storage data processing system. Background Art

[0002] With the advent of information technology and the data age, the demand for data storage and processing has shown explosive growth. In complex application scenarios such as high-performance computing, cloud computing, and data processing, computer memory systems, as the core link in data storage and processing, face severe challenges in terms of storage capacity, access speed, and resource utilization efficiency. Traditional storage systems have limitations in cache prefetching, storage management, data compression and decompression, and data security.

[0003] Existing cache systems mainly rely on static or predefined rules to pre-fetch data, which is difficult to adapt to complex data access patterns, resulting in cache misses, increasing data transmission delays between memory and processors, and affecting the overall performance of the system;

[0004] Modern computer storage systems contain multiple storage layers. There are significant performance differences between different storage media, which causes hot data to frequently access low-speed storage layers, and cold data to occupy high-speed memory resources, which reduces the system's response speed and wastes precious storage resources.

[0005] In a multi-level storage system, the data transmission process between storage media is vulnerable to attacks, there is a risk of data leakage and tampering, and dynamic security monitoring and encryption optimization cannot be achieved;

[0006] In a complex storage environment, system failures can lead to data loss and incompleteness, and the system cannot be quickly restored after a failure, increasing the risk of data loss;

[0007] In a multi-level storage system, the difference in data access latency between different storage levels significantly reduces the system's response speed, resulting in performance loss due to frequent cross-level access.

[0008] Therefore, those skilled in the art provide a computer memory data processing system to solve the problems raised in the above background technology. Summary of the invention

[0009] In view of the deficiencies of the prior art, the present invention provides a computer memory data processing system to solve the problems raised in the above background technology.

[0010] To achieve the above objectives, the present invention is implemented through the following technical solutions: A computer memory data processing system includes the following modules:

[0011] Intelligent cache prefetch module, which is used to predict future memory access addresses based on deep learning and reinforcement learning techniques, and load data into the cache in advance;

[0012] Heterogeneous storage management module, used to dynamically classify data based on data heat value and realize data migration between different storage media;

[0013] Storage consistency guarantee module, used to manage data consistency between different storage layers through the improved MESI protocol;

[0014] The data recovery module is used to recover unfinished write operations through the write-ahead log mechanism when the system fails;

[0015] A data compression and decompression module, used to compress the data in the memory and decompress the data when necessary;

[0016] Memory encryption and security module, used to encrypt transmitted and stored data;

[0017] The prefetch optimization and feedback module is used to adaptively adjust the cache prefetch strategy and dynamically optimize the prefetch depth and step size based on real-time performance feedback data.

[0018] Preferably, the intelligent cache prefetching module includes:

[0019] A data collection unit, used to collect memory access behavior information and extract spatial locality and temporal locality features;

[0020] The deep learning prediction model predicts future memory access addresses based on the LSTM network model. The prediction formula is:

[0021] a t+Δt =f LSTM (x t-k ,x t-k+1 ,…,x t |θ),

[0022] Among them, a t+Δt is the access address at the future time, f LSTM is a LSTM neural network, and θ is a model parameter.

[0023] Preferably, the intelligent cache prefetching module further comprises:

[0024] The reinforcement learning unit is used to optimize the prefetching strategy through the Ql earning algorithm and adjust the prefetching step size and depth. The Ql earning update formula is:

[0025]

[0026] Among them, Q(s t ,a t ) means in state s t Take action a t The expected reward is , α is the learning rate, and γ is the discount factor.

[0027] Preferably, the data compression and decompression module includes:

[0028] A compression algorithm unit, used for compressing the stored data by using a lossless compression algorithm;

[0029] The decompression algorithm unit is used to decompress the corresponding data block in real time according to the access frequency and data location when data is requested.

[0030] Preferably, the memory encryption and security module includes:

[0031] An encryption unit, used to encrypt data using a symmetric encryption algorithm when the data is transferred from the main memory to different storage levels;

[0032] The decryption unit is used to perform real-time decryption when data is returned from the storage medium to the main memory.

[0033] Preferably, the heterogeneous storage management module further includes:

[0034] Hot / cold data classifier, based on the heat of the data block h(D i ) to classify the data, and the heat calculation formula is:

[0035]

[0036] Among them, f(D i ) is the access frequency of the data block, t last (D i ) is the most recent access time, and α and β are adjustment parameters.

[0037] Preferably, the pre-fetch optimization and feedback module includes:

[0038] A performance feedback unit, used to monitor the performance indicators of the storage system in real time and input the feedback results into the cache prefetch module;

[0039] The adaptive optimization unit is used to dynamically adjust the prefetch strategy and optimize the prefetch depth and step size according to the feedback performance data.

[0040] Preferably, the data recovery module records data operation logs through a write-ahead log mechanism to ensure that unfinished write operations can be recovered through log backtracking when a failure occurs. The log update formula is:

[0041] Log NVM (Di ,t)=Log NVM (D i ,t-1)+write(D i ,t),

[0042] At time t, the data log recorded in the non-volatile memory is equal to the log state Log at the previous moment NVM (D i ,t-1) plus the current time for data block D i The write operation write(D i ,t).

[0043] Preferably, the computer memory data processing system further comprises:

[0044] The storage scheduler is used to dynamically adjust the cache pre-fetch strategy and data migration strategy according to the real-time performance indicators of the storage system.

[0045] Preferably, the compression algorithm unit includes a Huffman coding compression algorithm and an LZ77 compression algorithm:

[0046] The Huffman coding compression algorithm formula is a lossless compression algorithm that generates the optimal binary code based on the frequency distribution of data. Assume that there is a set of symbols S = {s1, s2, ..., s n}, the probability of occurrence is p1,p2,…,p n , the compression algorithm steps are as follows:

[0047] For each symbol s i , assign weight p i ;

[0048] By constructing a Huffman tree, the symbols are given the shortest code;

[0049] The final Huffman code length is:

[0050]

[0051] Among them, len(C(s i )) is the symbol s i The Huffman code length;

[0052] The LZ77 compression algorithm formula is a lossless compression algorithm based on a sliding window. Assume that the compressed data is a string X, the sliding window size is w, and the current symbol is x. i:t , whose compressed output is a triple:

[0053] C=(d,l,c),

[0054] Where d is the offset from the matching string, l is the length of the match, and c is the next unmatched character after the current match;

[0055] The decompression algorithm unit includes Huffman decompression algorithm and LZ77 decompression algorithm:

[0056] The Huffman decompression algorithm formula reads the prefix of the binary code and converts the code back to the original symbol sequence s1, s2, ..., s n , the specific decoding process is:

[0057] Read the compressed binary coded string and use the Huffman tree to match the prefix with the symbol;

[0058] Decode the binary string into the corresponding symbol s in sequence i ;

[0059] The decompression formula is: X = C -1 (L Huffman ),

[0060] Among them, C -1 Represents the inverse process of the Huffman tree, restoring the code to the original symbol;

[0061] The LZ77 decompression algorithm formula uses the compressed triples (d, l, c) to restore the data during the decompression process. For each triple, the matching substring x in the sliding window is located by d and l. i:t ;

[0062] Insert the substring into the decompressed output sequence and add the c character to the end;

[0063] The decompression formula is:

[0064] By iteratively decoding the triplets, the complete original data is restored.

[0065] The present invention provides a computer memory data processing system. It has the following beneficial effects:

[0066] 1. The present invention optimizes cache prefetching by introducing deep learning and reinforcement learning technologies. By predicting future memory access addresses in real time and loading data in advance, the system effectively reduces the cache miss rate, improves the cache hit rate, and significantly reduces the latency of memory access, so that the system can maintain more efficient data access performance in large-scale data processing scenarios.

[0067] 2. The present invention uses a heterogeneous storage management module to dynamically adjust the storage location of data between different storage media based on the popularity of the data. Hot data is preferentially stored in the high-speed storage layer, while cold data is migrated to the persistent storage layer. This can greatly improve the storage resource utilization of the system, reduce the storage and processing overhead caused by unnecessary data migration, and achieve a more balanced memory load.

[0068] 3. The present invention compresses the stored data through a lossless compression algorithm, significantly reducing the space occupied by the memory. At the same time, through an efficient decompression algorithm, it can quickly decompress and access data when data is requested, ensuring that the system does not affect the data access speed while ensuring storage efficiency. It is particularly beneficial for processing large-scale data sets and storage-intensive applications.

[0069] 4. The present invention introduces a memory encryption and security module, and adopts an encryption algorithm to encrypt and decrypt stored data, effectively preventing data leakage or tampering during transmission, thereby improving the overall data security of the system. In addition, by real-time monitoring of the performance indicators of the storage system, the pre-fetching strategy is dynamically adjusted to ensure that the pre-fetching depth and step size are optimized under different workloads and access modes, so that the system can adaptively adjust the pre-fetching behavior, reduce invalid pre-fetching operations, and further improve the system's operating efficiency and data processing capabilities.

[0070] 5. The write-before log mechanism of the present invention records unfinished write operations. When a system failure occurs, data can be restored by backtracking the log to ensure data integrity and persistence. This mechanism enhances the fault tolerance of the system and can quickly recover in the event of an unexpected power outage or system crash, reducing the risk of data loss and providing protection for high-reliability application scenarios.

[0071] 6. The present invention dynamically migrates hot / cold data through a heterogeneous storage management module, combined with cache pre-fetch optimization, to reduce access delays between different storage levels, so that the system can place high-frequency access data in a high-speed storage layer, while reducing the delay caused by frequent cross-layer access, and significantly improving the response speed of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a system framework diagram of the present invention;

[0073] Figure 2 It is a schematic diagram of a data compression and decompression module of the present invention;

[0074] Figure 3 It is a schematic diagram of the memory encryption and security module of the present invention;

[0075] Figure 4 It is a schematic diagram of the pre-fetch optimization and feedback module of the present invention. DETAILED DESCRIPTION

[0076] In order to make the technical personnel in the technical field understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a partial embodiment of the present invention, not a complete embodiment. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0077] The present invention is described in detail below in conjunction with the accompanying drawings:

[0078] Example:

[0079] Please refer to the attached Figure 1 -Attached Figure 4 , an embodiment of the present invention provides a computer memory data processing system, including the following modules:

[0080] Intelligent cache prefetch module, which is used to predict future memory access addresses based on deep learning and reinforcement learning techniques, and load data into the cache in advance;

[0081] Heterogeneous storage management module, used to dynamically classify data based on data heat value and realize data migration between different storage media;

[0082] Storage consistency assurance module, used to manage data consistency between different storage levels through the improved MES I protocol;

[0083] The data recovery module is used to recover unfinished write operations through the write-ahead log mechanism when the system fails;

[0084] A data compression and decompression module, used to compress the data in the memory and decompress the data when necessary;

[0085] Memory encryption and security module, used to encrypt transmitted and stored data;

[0086] The prefetch optimization and feedback module is used to adaptively adjust the cache prefetch strategy and dynamically optimize the prefetch depth and step size based on real-time performance feedback data.

[0087] Benefits of the Smart Cache Prefetch Module By combining deep learning and reinforcement learning technologies, the Smart Cache Prefetch Module can predict future memory access behaviors in real time and load required data into the cache in advance. The prefetch mechanism in the Smart Cache Prefetch Module reduces cache misses and reduces memory access latency, greatly improving the operating efficiency and response speed of the storage system in large-scale data processing scenarios.

[0088] Benefits of heterogeneous storage management module: Dynamically classify data based on the heat value of the data, intelligently store frequently accessed hot data in high-speed storage, and migrate infrequently used cold data to persistent storage media. This management method optimizes the utilization of storage resources, reduces unnecessary data migration overhead, and improves the overall performance and storage efficiency of the storage system;

[0089] Benefits of the storage consistency assurance module: Through the improved MES I protocol, data consistency between different storage levels is ensured, data inconsistency or redundancy between multi-level caches or storage media is avoided, data conflicts are effectively prevented, and data integrity and consistency of the system are ensured in high concurrency and multi-task execution;

[0090] Benefits of the data recovery module: If a system failure occurs, the data recovery module can quickly recover unfinished write operations through the write-before log mechanism, ensure data integrity and persistence, enhance the system's fault tolerance, enable the system to recover quickly from sudden failures, reduce the risk of data loss, and improve system reliability;

[0091] Benefits of the data compression and decompression module: By compressing the stored data, the space occupied by the memory is effectively reduced. It is particularly suitable for large-scale data storage and storage-intensive application scenarios. When needed, the system can quickly decompress the data and provide access services, ensuring efficient use of storage resources without affecting the speed and performance of data reading;

[0092] Benefits of the memory encryption and security module: It provides a strong encryption mechanism for data in storage and transmission, ensuring the security of data in storage media and transmission paths, preventing data leakage or tampering. It is particularly suitable for application scenarios of highly sensitive data processing, such as finance, medical and other fields, and can greatly improve the security of storage systems and the privacy protection capabilities of user data.

[0093] The benefits of the prefetch optimization and feedback module are that it adaptively adjusts the cache prefetch strategy through real-time performance feedback, dynamically optimizes the prefetch depth and step size, and enables the system to maintain the best prefetch efficiency under different workloads and access modes, further reducing invalid prefetches and improving the overall data processing capability and system operating efficiency.

[0094] The intelligent cache prefetch module includes:

[0095] A data collection unit, used to collect memory access behavior information and extract spatial locality and temporal locality features;

[0096] The deep learning prediction model predicts future memory access addresses based on the LSTM network model. The prediction formula is:

[0097] a t+Δt =f LSTM(x t-k ,x t-k+1 ,…,x t |θ),

[0098] Among them, a t+Δt is the access address at the future time, f LSTM is a LSTM neural network, and θ is a model parameter.

[0099] The benefits of the data acquisition unit are that it can capture memory access patterns, help identify data access hotspots and commonly used data, and enable the system to pre-fetch data more intelligently, provide more accurate data features, and help improve the accuracy of subsequent prediction models, while reducing unnecessary data loading and improving cache hit rates;

[0100] What are the benefits of deep learning prediction models? By modeling the timing data of memory access through the LSTM network, it can more accurately predict future memory access behavior, greatly improve the accuracy of cache prefetching, and predict future access addresses in advance and load data. The system greatly reduces cache misses and improves memory access efficiency. At the same time, the LSTM model can handle complex timing dependencies and is suitable for multi-tasking and multi-threaded memory access scenarios in large-scale data processing scenarios, ensuring that the system still runs efficiently in a dynamic environment.

[0101] The intelligent cache prefetch module further includes:

[0102] The reinforcement learning unit is used to optimize the prefetching strategy through the Q-learning algorithm and adjust the prefetching step size and depth. The Q-learning update formula is:

[0103]

[0104] Among them, Q(s t ,a t ) means in state s t Take action a t The expected reward is , α is the learning rate, and γ is the discount factor.

[0105] The advantage of the reinforcement learning unit is that it can perform adaptive learning based on the real-time feedback of the system, continuously optimize the prefetch step size and depth, and ensure that the cache prefetch strategy is always in the optimal state. The system faces frequent data access or large-scale continuous data access, and can adaptively adjust the strategy to improve the overall prefetch efficiency.

[0106] Through the Ql earning algorithm, the system can optimize prefetching behavior after real-time evaluation, reduce unnecessary prefetching operations, avoid loading redundant data, reduce cache resource waste, and improve cache hit rate;

[0107] Reinforcement learning technology can cope with diverse workloads and access patterns. In high-concurrency environments, complex data processing, and dynamically changing application scenarios, the system can self-learn and continuously optimize its caching strategy to maintain the efficiency of the system.

[0108] The data compression and decompression modules include:

[0109] A compression algorithm unit, used for compressing the stored data by using a lossless compression algorithm;

[0110] The decompression algorithm unit is used to decompress the corresponding data block in real time according to the access frequency and data location when data is requested.

[0111] Benefits of the compression algorithm unit: The storage data is compressed by a lossless compression algorithm, thereby reducing the space occupied by the memory. Lossless compression means that no information will be lost during the compression and decompression process, and the original data can be fully restored. By compressing the stored data, the system significantly reduces the space occupied by the memory, especially when storing a large amount of data or a large-scale data set, which can greatly improve the utilization rate of storage resources. The amount of compressed data is reduced, saving storage space, and can reduce the bandwidth occupied when data is transmitted between storage layers, thereby improving data transmission efficiency and reducing delays during cross-layer transmission.

[0112] Benefits of the decompression algorithm unit: By deterministically decompressing data based on its access frequency and location, the system can prioritize decompression and provide frequently accessed data, ensuring rapid response and efficient processing of key data by the system, greatly improving data access speed; it can intelligently determine the priority of different data and dynamically decompress the most important data blocks, thereby improving the system's real-time response capability; it ensures that the system decompresses data when needed without wasting system resources on decompressing infrequently used or low-priority data blocks, reducing unnecessary computing overhead and improving system efficiency. While making efficient use of storage space, the system maintains the integrity and accuracy of the original data, allowing the system to take into account both storage efficiency and performance.

[0113] The memory encryption and security module includes:

[0114] An encryption unit, used to encrypt data using a symmetric encryption algorithm when the data is transferred from the main memory to different storage levels;

[0115] The decryption unit is used to perform real-time decryption when data is returned from the storage medium to the main memory.

[0116] The encryption unit encrypts data through a symmetric encryption algorithm, greatly improving the data security in the storage system. It is particularly suitable for application scenarios that store sensitive data, such as finance, medical care, government, and other fields. Data is vulnerable to malicious attacks during transmission between different storage levels. The encryption unit can effectively prevent these attacks, prevent data from being tampered with, forged, or leaked, and ensure the integrity and correctness of data during transmission. The storage level in the system is complicated, and data is frequently transmitted between multi-level storage devices. The encryption unit provides security protection for cross-layer transmission, ensuring that data transmitted between different storage media is not subject to security threats.

[0117] The benefits of the decryption unit are that it can decrypt data in real time when the data is returned from the storage medium to the main memory, ensuring that the decryption operation does not introduce additional delays, ensuring that the system can efficiently access and process encrypted sensitive data, and maintaining the high performance of the system; when the data storage medium returns to the main memory, the decryption unit can accurately decrypt and restore the data, ensuring that the data maintains its integrity and privacy during transmission and storage, which is crucial for processing data that requires high privacy protection; decrypting data transmitted back to the main memory when necessary, and avoiding decryption operations on non-essential data, saving computing resources and improving the overall operating efficiency of the system; through real-time decryption and encryption processing, the system can ensure full encryption during data storage and transmission, comply with strict security and privacy protection regulations, and provide compliance support for enterprises and organizations.

[0118] The heterogeneous storage management module further includes:

[0119] Hot / cold data classifier, based on the heat of the data block h(D i ) to classify the data, and the heat calculation formula is:

[0120]

[0121] Among them, f(D i ) is the access frequency of the data block, t last (D i ) is the most recent access time, and α and β are adjustment parameters.

[0122] The heat of a data block is calculated by combining the access frequency and the most recent access time of the data block. A high heat value indicates that the data block is frequently accessed and is high-frequency "hot data". On the contrary, it is "cold data". Its classification helps the system identify that data should be stored in high-speed memory first, and data can be migrated to a lower-speed persistent storage layer.

[0123] The system can dynamically adjust the storage location of data among different storage media according to the heat value of each data block. Hot data is preferentially stored in high-speed storage to speed up access, while cold data is migrated to low-speed but large-capacity storage to optimize storage resource utilization.

[0124] α and β are adjustment parameters that control the relative importance of access frequency and recent access time in calculating heat. By adjusting the parameters, the system can set different priorities according to different application scenarios. For example, in some scenarios, the timeliness of recently accessed data is more important, while in other scenarios, the long-term access frequency of data is more worthy of attention.

[0125] The prefetch optimization and feedback module includes:

[0126] A performance feedback unit, used to monitor the performance indicators of the storage system in real time and input the feedback results into the cache prefetch module;

[0127] The adaptive optimization unit is used to dynamically adjust the prefetch strategy and optimize the prefetch depth and step size according to the feedback performance data.

[0128] Benefits of the Performance Feedback Unit: By real-time monitoring of the system's performance indicators, the Performance Feedback Unit can promptly capture performance bottlenecks or efficiency issues in the system. Real-time monitoring enables the system to quickly adjust the pre-fetching strategy to ensure that the system is always in an efficient operating state. The Performance Feedback Unit feeds back the monitoring results to the cache pre-fetching module, so that the optimization of the pre-fetching strategy is based on actual operating data and performance conditions. The pre-fetching behavior can adaptively respond to the current load and access mode of the system to ensure optimal resource utilization. Through continuous feedback and monitoring, the Performance Feedback Unit can help identify invalid pre-fetching and make adjustments, reduce system resource waste, and improve overall resource utilization.

[0129] The benefits of the adaptive optimization unit are that it can dynamically adjust the prefetch strategy according to the actual performance of the storage system, avoiding the use of static, predefined strategies. The system can flexibly adjust the prefetch step and depth to adapt to changes in data access and achieve more efficient data prefetching; by optimizing the prefetch depth and step, the system can more accurately predict the data needed in the future and load it into the cache in advance, thereby improving the cache hit rate, reducing the delay in memory access, and increasing the speed of data access; the adaptive optimization unit can promptly identify excessive invalid prefetch operations through performance feedback, and correct the prefetch strategy to avoid loading unnecessary data; through real-time adaptive optimization, the system can better cope with changes in different workloads, avoid performance degradation due to sudden changes in data access patterns, and ensure that the system can quickly respond to changes in demand in various scenarios.

[0130] The data recovery module records data operation logs through the write-before log mechanism to ensure that incomplete write operations can be recovered through log backtracking in the event of a failure. The log update formula is:

[0131] Log NVM (D i ,t)=Log NVM (Di ,t-1)+write(D i ,t),

[0132] At time t, the data log recorded in the non-volatile memory is equal to the log state Log at the previous moment NVM (D i ,t-1) plus the current time for data block D i The write operation write(D i ,t).

[0133] Through the write-before log mechanism, the data recovery module can ensure that when a failure occurs, the system can use log backtracking to recover unfinished write operations, ensuring data integrity, persistence and security. It not only enhances the system's fault tolerance, can quickly restore system operation, and reduce downtime, but also significantly reduces the risk of data loss, and is particularly suitable for high-concurrency, data-sensitive or critical application scenarios.

[0134] The computer memory data processing system further comprises:

[0135] The storage scheduler is used to dynamically adjust the cache pre-fetch strategy and data migration strategy according to the real-time performance indicators of the storage system.

[0136] The compression algorithm unit includes Huffman coding compression algorithm and LZ77 compression algorithm:

[0137] The Huffman coding compression algorithm formula is a lossless compression algorithm that generates the optimal binary code based on the frequency distribution of data. Assume that there is a set of symbols S = {s1, s2, ..., s n}, the probability of occurrence is p1,p2,…,p n , the compression algorithm steps are as follows:

[0138] For each symbol s i , assign weight p i ;

[0139] By constructing a Huffman tree, the symbols are given the shortest code;

[0140] The final Huffman code length is:

[0141]

[0142] Among them, len(C(s i )) is the symbol s i The Huffman code length;

[0143] The LZ77 compression algorithm formula is a lossless compression algorithm based on a sliding window. Assume that the compressed data is a string X, the sliding window size is w, and the current symbol is x. i:t, whose compressed output is a triple:

[0144] C=(d,l,c),

[0145] Where d is the offset from the matching string, l is the length of the match, and c is the next unmatched character after the current match;

[0146] The decompression algorithm unit includes Huffman decompression algorithm and LZ77 decompression algorithm:

[0147] The Huffman decompression algorithm formula reads the prefix of the binary code and converts the code back to the original symbol sequence s1, s2, ..., s n , the specific decoding process is:

[0148] Read the compressed binary coded string and use the Huffman tree to match the prefix with the symbol;

[0149] Decode the binary string into the corresponding symbol s in sequence i ;

[0150] The decompression formula is: X = C -1 (L Huffman ),

[0151] Among them, C -1 Represents the inverse process of the Huffman tree, restoring the code to the original symbol;

[0152] LZ77 decompression algorithm formula, the decompression process uses the compressed triples (d, l, c) to restore the data, and for each triple, locate the matching substring x in the sliding window through d and l i:t ;

[0153] Insert the substring into the decompressed output sequence and add the c character to the end;

[0154] The decompression formula is:

[0155] By iteratively decoding the triplets, the complete original data is restored.

[0156] The function of the Huffman coding compression algorithm formula is to assign short codes to high-frequency symbols. Huffman coding can achieve more efficient compression, reduce the space occupied by data during storage and transmission, avoid any information loss during compression and decompression, and ensure that the original data can be fully restored;

[0157] The function of the LZ77 compression algorithm formula is to reduce redundancy and achieve compression by referencing previously appeared strings. The function of LZ77 is to process data with repeated patterns. Its compression strategy can find repeated patterns in the data and replace them, thereby effectively reducing the amount of data. It also reduces redundant data and improves compression efficiency by referencing matching strings that have already appeared in the sliding window.

[0158] The function of the Huffman decompression algorithm formula is to restore the binary code to the original symbol sequence through the constructed Huffman tree, ensure that the data compressed by the Huffman code can be accurately decoded and restored to the original data, and use the Huffman tree to match the prefixes one by one and convert them back to the original symbols to ensure that the data is completely and correctly decompressed. After decompression, the original data will not lose any information and the original symbol sequence will be completely restored;

[0159] The function of the LZ77 decompression algorithm formula is to decompress the data compressed by triples into the original string. The LZ77 decompression algorithm can effectively restore the data compressed by referencing repeated patterns in the sliding window, ensuring the integrity of the original data. It can efficiently decompress large-scale data sets, and is especially suitable for compression and decompression of text or log data.

[0160] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A computer memory data processing system, characterized in that: Includes the following modules: Intelligent cache prefetch module, which is used to predict future memory access addresses based on deep learning and reinforcement learning techniques, and load data into the cache in advance; Heterogeneous storage management module, used to dynamically classify data based on data heat value and realize data migration between different storage media; Storage consistency guarantee module, used to manage data consistency between different storage layers through the improved MESI protocol; The data recovery module is used to recover unfinished write operations through the write-ahead log mechanism when the system fails; A data compression and decompression module, used to compress the data in the memory and decompress the data when necessary; Memory encryption and security module, used to encrypt transmitted and stored data; The prefetch optimization and feedback module is used to adaptively adjust the cache prefetch strategy and dynamically optimize the prefetch depth and step size based on real-time performance feedback data.

2. A computer memory data processing system according to claim 1, characterized in that: The intelligent cache prefetching module comprises: A data collection unit, used to collect memory access behavior information and extract spatial locality and temporal locality features; The deep learning prediction model predicts future memory access addresses based on the LSTM network model. The prediction formula is: a t+Δt =f LSTM (x t-k ,x t-k+1 ,…,x t |θ), Among them, a t+Δt is the access address at the future time, f LSTM is a LSTM neural network, and θ is a model parameter.

3. A computer memory data processing system according to claim 1 or 2, characterized in that: The intelligent cache prefetching module further comprises: The reinforcement learning unit is used to optimize the prefetching strategy through the Q-learning algorithm and adjust the prefetching step size and depth. The Q-learning update formula is: Among them, Q(s t ,a t ) means in state s t Take action a t The expected reward is , α is the learning rate, and γ is the discount factor.

4. A computer memory data processing system according to claim 1, characterized in that: The data compression and decompression module comprises: A compression algorithm unit, used for compressing the stored data by using a lossless compression algorithm; The decompression algorithm unit is used to decompress the corresponding data block in real time according to the access frequency and data location when data is requested.

5. A computer memory data processing system according to claim 1, characterized in that: The memory encryption and security module includes: An encryption unit, used to encrypt data using a symmetric encryption algorithm when the data is transferred from the main memory to different storage levels; The decryption unit is used to perform real-time decryption when data is returned from the storage medium to the main memory.

6. A computer memory data processing system according to any one of claims 1 to 5, characterized in that: The heterogeneous storage management module further includes: Hot / cold data classifier, based on the heat of the data block h(D i ) to classify the data, and the heat calculation formula is: Among them, f(D i ) is the access frequency of the data block, t last (D i ) is the most recent access time, and α and β are adjustment parameters.

7. A computer memory data processing system according to claim 6, characterized in that: The pre-fetch optimization and feedback module includes: A performance feedback unit, used to monitor the performance indicators of the storage system in real time and input the feedback results into the cache prefetch module; The adaptive optimization unit is used to dynamically adjust the prefetch strategy and optimize the prefetch depth and step size according to the feedback performance data.

8. A computer memory data processing system according to claim 1, characterized in that: The data recovery module records data operation logs through a write-before log mechanism to ensure that unfinished write operations can be recovered through log backtracking when a failure occurs. The log update formula is: Log NVM (D i ,t)=Log NVM (D i ,t-1)+write(D i ,t), At time t, the data log recorded in the non-volatile memory is equal to the log state Log at the previous moment NVM (D i ,t-1) plus the current time for data block D i The write operation write(D i ,t).

9. A computer memory data processing system according to any one of claims 1 to 8, characterized in that: The computer memory data processing system further comprises: The storage scheduler is used to dynamically adjust the cache pre-fetch strategy and data migration strategy according to the real-time performance indicators of the storage system.

10. A computer memory data processing system according to claim 4, characterized in that: The compression algorithm unit includes Huffman coding compression algorithm and LZ77 compression algorithm: The Huffman coding compression algorithm formula is a lossless compression algorithm that generates the optimal binary code based on the frequency distribution of data. Assume that there is a set of symbols S = {s1, s2, ..., s n }, the probability of occurrence is p1,p2,…,p n , the compression algorithm steps are as follows: For each symbol s i , assign weight p i ; By constructing a Huffman tree, the symbols are given the shortest code; The final Huffman code length is: Among them, len(C(s i )) is the symbol s i The Huffman code length; The LZ77 compression algorithm formula is a lossless compression algorithm based on a sliding window. Assume that the compressed data is a string X, the sliding window size is w, and the current symbol is x. i:t , whose compressed output is a triple: C=(d,l,c), Where d is the offset from the matching string, l is the length of the match, and c is the next unmatched character after the current match; The decompression algorithm unit includes Huffman decompression algorithm and LZ77 decompression algorithm: The Huffman decompression algorithm formula reads the prefix of the binary code and converts the code back to the original symbol sequence s1, s2, ..., s n , the specific decoding process is: Read the compressed binary coded string and use the Huffman tree to match the prefix with the symbol; Decode the binary string into the corresponding symbol s in sequence i ; The decompression formula is: X = C -1 (L Huffman ), Among them, C -1 Represents the inverse process of the Huffman tree, restoring the code to the original symbol; The LZ77 decompression algorithm formula uses the compressed triples (d, l, c) to restore the data during the decompression process. For each triple, the matching substring x in the sliding window is located by d and l. i:t ; Insert the substring into the decompressed output sequence and add the c character to the end; The decompression formula is: By iteratively decoding the triplets, the complete original data is restored.

Images