A data management method for semiconductor storage
By introducing dynamic cache mapping tables, multi-version concurrency control and delayed write correction algorithms in semiconductor storage systems, data consistency problems under high concurrent access and frequent write operations are solved, and the system efficiency and reliability improvement is achieved.
Patent Information
- Application Number
- CN202411244918.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-06
AI Technical Summary
In semiconductor storage systems, the existing address mapping mechanism is prone to cause data consistency problems under high concurrent access and frequent write operations, resulting in a ‘write-travel’ phenomenon, affecting system performance and reliability.
Introducing dynamic cache mapping table, multi-version concurrency control and delayed write correction algorithm. The dynamic cache mapping table dynamically adjusts the mapping relationship according to the access mode. Multi-version concurrency control allows multiple versions of data to be retained. The delayed write correction algorithm uses delayed write and verification ensures consistency and correctness of the data.
It significantly improves the efficiency and reliability of the semiconductor storage system, prevents the phenomenon of "write time travel", ensures the consistency and correctness of data in high concurrency and frequent write operation scenarios, and improves the performance and stability of the system.
Smart Images

Figure CN118760623B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of semiconductor data storage management, and more particularly, to a data management method for semiconductor storage. Background Art
[0002] In semiconductor memory systems, data management is a key part of ensuring data reliability and storage efficiency. In these memories, data management methods involve data access, addressing, and allocation to ensure that data can be effectively and accurately read from or written to the storage medium. In current memory systems, common data management methods include data caching, data prefetching, address mapping, and error correction code (ECC). These methods are mainly used to improve data access speed, improve system response time, and ensure data integrity. For example, in high-performance disk storage units of network attached storage (NAS) and storage area network (SAN) storage systems in enterprise-level data centers, disk storage units must handle a large number of concurrent access requests from multiple application virtual machines or user groups. Data is usually managed through a cache mechanism. The cache can temporarily store frequently accessed data to reduce direct access to the disk and improve the performance of the overall system. The addressing mechanism is also crucial, especially in large storage systems. Correct The addressing method can significantly improve the data reading and writing speed and system stability. The existing addressing technologies include sequential addressing, direct addressing and logical block addressing (LBA). Logical block addressing is a method of replacing physical addresses with logical blocks, which effectively simplifies the complexity of data access. However, there are still some challenges in address mapping and data consistency management in memory systems. Although logical block addressing (LBA) in the existing address mapping mechanism simplifies the complexity of data access, it is easy to cause data consistency problems during multi-tasking or concurrent access, especially under high concurrent access, especially frequent write operations. Specifically, when multiple processes or threads access the storage unit at the same time, if the mapping table and data cache cannot be effectively managed, it will lead to the phenomenon of "write-through" of data, that is, the new data in the cache is not refreshed to the storage medium in time, and the old data is still returned when reading. This situation not only affects the performance of the system, but also causes unpredictable errors in the data, thereby affecting the reliability of the system. Summary of the invention
[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides a data management method for semiconductor storage, which significantly improves the efficiency and reliability of the semiconductor storage system by introducing a dynamic cache mapping table, multi-version concurrency control and delayed write correction algorithm.
[0004] In enterprise-level storage systems, multiple applications or processes access the same unit at the same time. For example, in a virtualized environment, different virtual machines read and write the same logical block address at the same time. The newly written data fails to be refreshed to the physical medium in time, and the read operation returns the old data, which causes data consistency problems. This is particularly fatal in database, file system or critical business system applications, which will lead to data errors, business interruptions or data corruption. In the high-load environment of the data center, the disk storage unit often handles a large number of concurrent access requests. This is particularly common in applications such as large databases, real-time data analysis, virtual desktop infrastructure and high-frequency trading. Frequent write operations will cause the data in the cache to fail to be written to the disk in time, resulting in a "write-through" phenomenon, that is, the read operation reads expired data. For these applications, any data delay or inconsistency will cause disk storage performance degradation or failure. Although logical block addressing simplifies data access, in a complex enterprise storage environment, when different tasks or users request data at the same time, the management of the mapping table becomes more complicated. If the mapping table cannot be managed effectively, it will lead to mapping relationship errors, affecting data accuracy and storage efficiency. This is particularly obvious in cloud computing environments, hybrid storage architectures or multi-tenant data centers, which will lead to unforeseen data storage errors and data access delays.
[0005] In order to solve the technical problems existing in the above application scenarios and achieve the above objectives, the present invention provides the following technical solutions:
[0006] A data management method for semiconductor storage comprises the following steps:
[0007] Step 1, initialize the dynamic cache mapping table (DCMT): When the system is initialized, create a dynamic cache mapping table (DCMT) for storing the mapping of logical block addresses to physical block addresses. This dynamic cache mapping table (DCMT) is dynamically adjusted as the access mode changes to improve the cache hit rate and data access efficiency;
[0008] Step 2, Multi-version Concurrency Control (MVCC): When writing data, introduce multi-version concurrency control (MVCC). Multi-version concurrency control (MVCC) allows the old version of data to be retained while writing, responding to concurrent access requests and avoiding the "write-through" phenomenon;
[0009] Step 3, Delayed Write Correction (DCW): The Delayed Write Correction (DWC) algorithm is used to avoid data inconsistency caused by frequent write operations. The data is written from the cache to the storage medium by delaying, and the data is corrected and verified during the delay period to ensure the consistency and correctness of the data when it is written to the medium.
[0010] Step 4, cache elimination and consistency check: Use the least recently used strategy (LRU) to eliminate cache and perform consistency check before elimination to ensure that all data has been correctly written to the storage medium when it is removed from the cache.
[0011] As a further solution of the present invention, in step 1, when the system is initialized, a dynamic cache mapping table (DCMT) for storing mappings from logical block addresses to physical block addresses is created. The specific implementation process of the dynamic cache mapping table (DCMT) being dynamically adjusted as the access mode changes is as follows:
[0012] Step 11, feature collection and analysis: When the system is initialized, start collecting the access frequency of the logical block address (LBA) (recording the number of accesses and time intervals for each LBA), historical access patterns (identifying and recording the access patterns of each LBA in different time periods, including peak time and access periodicity), and data types (recording the data type stored in the LBA, such as random data or sequential data, and its impact on system performance), analyze these collected features, and generate an initial feature matrix (this matrix is used to calculate the dynamic adjustment factor to help determine which LBAs should be stored in the cache first);
[0013] Step 12, creating an initial mapping table: based on the initial feature matrix, a mapping table from logical block address (LBA) to physical block address (PBA) is preliminarily generated. The mapping table is generated using a high access frequency priority rule. Logical block addresses (LBA) with high access frequency are preferentially mapped to faster physical storage units. According to the data type of the logical block address (LBA), a suitable data storage unit is matched (for sequential data, a unit with higher sequential read performance is selected), and an initial dynamic adjustment factor is generated for each logical block address (LBA);
[0014] Step 13, dynamic adjustment mechanism: When the system is running, the access mode, system load and feature changes of each logical block address (LBA) are continuously monitored (the monitored data will be used to update the feature matrix and system load matrix in real time), and the adjustment factor of each logical block address (LBA) is dynamically adjusted based on the latest monitoring data. The update frequency of the adjustment factor is set according to the fluctuation of the system load and access frequency (for example, adjustment is performed at regular intervals or when a certain threshold is reached). According to the updated adjustment factor, the mapping of the logical block address (LBA) to the physical block address (PBA) is recalculated (if the adjustment factor of a logical block address (LBA) increases significantly, it will be mapped to a storage unit with higher performance to improve the overall system performance);
[0015] Step 14, data migration and synchronization: When updating the mapping table, the logical block address (LBA) changes, and data migration is performed. The data is migrated from the old address to the new address, and the data integrity and consistency of the migration process are guaranteed. After the mapping table is updated, the updated mapping relationship is synchronized to the cache and storage controller (to ensure that all parts of the system are consistent with the latest mapping relationship);
[0016] Step 15, system verification and optimization: Regularly perform consistency checks on the dynamic cache mapping table to ensure the correctness of the mapping relationship, especially in scenarios with high concurrent access and frequent write operations, to prevent the "write-through" phenomenon. Based on feedback from system operation, continuously optimize the calculation rules of the adjustment factor and the update mechanism of the mapping table to further improve the system's response speed and reliability.
[0017] As a further solution of the present invention, in step 13, the mapping of the logical block address (LBA) to the physical block address (PBA) is recalculated according to the updated adjustment factor, and the mapping function is:
[0018] ;
[0019] Where: is the logical block address, is the mapping function value of the logical block address, is the physical block address corresponding to the logical block address, is the dynamic adjustment factor, obtained by calculation, Is the current timestamp.
[0020] As a further solution of the present invention, in the mapping function formula in step 13, the three-dimensional tensor representing the composite relationship between the feature matrix, the load matrix, and the priority matrix is multiplied with the feature matrix in the first dimension, and then multiplied with the matrix obtained by the element-by-element operation of the system load matrix index in the second dimension, and the obtained matrix is added to the matrix obtained by the sine element-by-element operation of the mapping priority matrix, and the obtained matrix is used as the dynamic adjustment factor. The formula of the dynamic adjustment factor is:
[0021] ;
[0022] Where: is a three-dimensional tensor representing the interaction between the feature matrix, system load matrix, and priority matrix corresponding to the current timestamp. is a feature matrix used to capture the access frequency, historical access pattern, and data type dimension characteristics of each logical block address (LBA). The feature dimension is , the feature matrix dimension is ,in, is the number of LBAs, is the number of feature dimensions, , are the product operations of tensor and matrix in the first and second dimensions respectively. is the system load matrix, which represents the system at timestamp The load condition of , is the mapping priority matrix, which indicates the mapping priority of each LBA, and its dimension is , , , They are the system load adjustment coefficient, mapping priority adjustment coefficient and frequency adjustment coefficient. for The matrix obtained after performing element-by-element operation on index e, for The matrix resulting from the element-by-element operation of sine.
[0023] is a coefficient used to control the degree to which the system load affects the dynamic adjustment factor. It determines how the adjustment factor should respond when the system load is high. The higher the load, The larger the value, the slower the growth rate of the adjustment factor, which prevents system resources from being overly concentrated on certain LBAs, thereby avoiding system bottlenecks. The value of is usually derived from statistical analysis of the following data sources: (1) Real-time system monitoring data: including CPU utilization, memory usage, I / O operation number, etc., which directly reflects the current load of the system; (2) Historical load analysis: by analyzing the load pattern of historical data, the appropriate load for the system is calculated. (3) Performance test results: Based on system performance testing, determine the optimal value to balance performance and resource utilization.
[0024] It is used to control the impact of mapping priority on the adjustment factor. It determines to what extent the system should consider the priority of LBA when allocating resources. High-priority LBA will get more resource allocation to ensure that these important data blocks can be accessed quickly. The value of can be determined by the following data sources: (1) Data importance index: According to the system's assessment of the importance of data blocks (such as key business data, real-time data, etc.), different priorities are assigned, which in turn affects (2) User settings or policies: System administrators or users can manually adjust the priority of LBAs according to business needs to reflect current business priorities; (3) Access pattern analysis: By analyzing the historical access patterns of LBAs, their importance in future accesses is calculated, so as to set reasonable value.
[0025] It is a coefficient used to control the periodic effect of access frequency on the adjustment factor. It helps capture the periodic characteristics of LBA access in the system and adapt to the access rules of different LBAs. A value of 0 means that the adjustment factor will more sensitively reflect periodic changes in access frequency. The value of is based on the following data sources: (1) Access frequency statistics: By monitoring the access frequency of each LBA in real time, its periodic characteristics are determined, and the appropriate (2) Historical access data: Analyze long-term access records, identify LBA access periodicity patterns, and set them accordingly. ; (3) Specific requirements of usage scenarios: In some specific scenarios (such as real-time data analysis or high-frequency trading systems), The value will be adjusted according to the needs of the scenario to better adapt to high-frequency and periodic access patterns.
[0026] As a further solution of the present invention, in step 2, the specific implementation process of multi-version concurrency control (MVCC) includes:
[0027] Step 21, data version management: In the storage system, each logical block address (LBA) pair will be written multiple times at different time points. A new data version is generated for each write operation, while the old data version is retained. A unique version number is set for each data version using the version management function to identify the data generation time. The data version of each logical block address (LBA) is recorded in a version chain, which is sorted by timestamp, with the latest version at the end of the chain.
[0028] Step 22, data read operation: When the system needs to read data, the read operation selects a suitable version from the version chain according to the data read function based on the current system time and transaction timestamp to read, so as to ensure that the read operation will not be affected by concurrent write operations;
[0029] Step 23, data writing operation: When performing the data writing operation, the system does not directly overwrite the existing data version, but uses the data writing function to generate a new version of the data and marks it with the current timestamp (so that even in the case of concurrent writing, the old version of the data is still available to ensure data consistency and integrity);
[0030] Step 24, version garbage collection: regular cleaning based on the version recycling function or intelligent cleaning based on the access pattern to release the storage space occupied by the version garbage.
[0031] As a further solution of the present invention, in step 21 of the specific implementation process of step 2, the version management function is:
[0032] ;
[0033] Where: is the data version index, The timestamp for generating the data version. For When writing data to the logical block address (LBA), For The version management function value corresponding to the data written to the logical block address (LBA) when the
[0034] As a further solution of the present invention, in step 22 of the specific implementation process of step 2, the formula of the data reading function is:
[0035] ;
[0036] Where: is the timestamp of the current transaction, For all less than or equal to Select the maximum timestamp Data version To read, For the current time point The latest version of the data available.
[0037] As a further solution of the present invention, in step 23 of the specific implementation process of step 2, the formula of the data writing function is:
[0038] ;
[0039] Where: The time when the current new data version is generated. The generated new data version contains the new data and timestamp .
[0040] As a further solution of the present invention, in step 24 of the specific implementation process of step 2, the version recovery function is:
[0041] ;
[0042] Where: Remember the recycling operation for the version to delete the old version data that is no longer needed. The time threshold is set, and versions below this timestamp are recycled.
[0043] As a further solution of the present invention, in step 3, the specific implementation process of the delayed write correction algorithm is:
[0044] Step 31, cache data marking and delay queue management: when data is written into the cache, each data block is marked as uncommitted, indicating that the data has not been written into the physical storage medium. The data is put into the delay write queue. After the data enters the delay write queue, the queue is sorted according to the time sequence of the write request, and the initial write time of each data block is recorded;
[0045] Step 32, delayed writing and data verification: after the data enters the delay queue, the time when the data is written into the physical storage medium is controlled according to the preset delay time. If the data does not change during the delay period, it is written according to the planned time. If the data changes, the delay time is recalculated and verified. During the delay period, the system verifies the data in the delay queue to detect whether there is a data conflict or inconsistency. If a conflict or data error is found, the data is corrected to ensure that the data finally written is correct.
[0046] Step 33, submission and consistency check: When the data is finally written to the physical storage medium, the system writes the data in the delay queue to the physical storage medium. After writing, the data status changes from unsubmitted to submitted. After the data is written to the medium, the system performs a consistency check again to ensure that no errors or loss occur during the writing process. If the check passes, the data status changes to submitted.
[0047] As a further solution of the present invention, in step 4, the specific implementation process of using the least recently used strategy (LRU) to perform cache elimination is:
[0048] Step 41, cache access record and sorting: the system maintains each access record for each logical block address (LBA) in the cache. Each time a data block is accessed, the access timestamp of the data block is updated. All data blocks in the cache are sorted according to the access timestamp, with the most recently accessed blocks placed at the front and the least recently accessed blocks placed at the end. In this way, the data blocks in the cache are arranged in order of access frequency and time.
[0049] Step 42, cache elimination strategy execution: When the cache space is insufficient and space needs to be freed, the system selects the data block that has not been accessed for the longest time for elimination through sorting, that is, selects the last data block in the sorted list for removal, and the selected data block is removed from the cache. If the content of the data block has not been written to the storage medium, it will be written to the storage medium before removal to ensure that the data is not lost.
[0050] Step 43, data consistency check: when executing cache elimination, the system performs data consistency check to ensure that the removed data block has been correctly written to the storage medium. If the consistency check fails, the system will prevent the elimination of the data block until the problem is resolved.
[0051] Step 44, cache update and synchronization: After the data block in the cache is removed, the space in the cache is released, and the system can continue to cache new data blocks. At this time, the cache records and sorting lists will also be updated accordingly, and the latest cache status will be synchronized to other modules of the system to ensure that the entire system synchronizes the cache status.
[0052] Compared with the prior art, the technical effect of the present invention is as follows: the present invention significantly improves the data management efficiency and reliability of disk storage units in network attached storage and storage area networks in enterprise-level storage system data centers by introducing dynamic cache mapping tables, multi-version concurrency control and delayed write correction algorithms. The dynamic cache mapping table can dynamically adjust the mapping relationship according to changes in access patterns, improve cache hit rate and data access speed, and is particularly suitable for high-concurrency access scenarios. The multi-version concurrency control mechanism allows multiple versions of data to be retained to prevent the "write-through" phenomenon and ensure data consistency under concurrent write operations. The delayed write correction algorithm reduces the risk of data inconsistency caused by frequent write operations by delaying writes and verification, and ensures the correctness of data when writing to the medium. The least recently used strategy is combined with consistency verification to ensure that cached data has been correctly stored on the medium before being removed. This method enhances the system's data processing capabilities under high load and high concurrency conditions, and improves the system's performance, data reliability and stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the present invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described technical solutions are only part of the present invention but not all of it. Based on the content of the present invention, all other technical solutions obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0055] The present invention proposes a data management method for semiconductor storage, which significantly improves the data management efficiency and reliability of disk storage units in network attached storage and storage area networks in enterprise-level storage system data centers by introducing a dynamic cache mapping table, multi-version concurrency control and a delayed write correction algorithm. The dynamic cache mapping table can dynamically adjust the mapping relationship according to the change of the access mode, improve the cache hit rate and data access speed, and is particularly suitable for high-concurrency access scenarios. The multi-version concurrency control mechanism allows multiple versions of data to be retained to prevent the "write-through" phenomenon and ensure data consistency under concurrent write operations. The delayed write correction algorithm reduces the risk of data inconsistency caused by frequent write operations by delaying writing and verification, and ensures the correctness of data when writing to the medium. The least recently used strategy is combined with consistency verification to ensure that the cached data has been correctly stored on the medium before being removed. The method enhances the data processing capability of the system under high load and high concurrency conditions, and improves the performance, data reliability and stability of the system.
[0056] like Figure 1 As shown, the method proposed by the present invention comprises the following steps:
[0057] Step 1, initialize the dynamic cache mapping table (DCMT): When the system is initialized, create a dynamic cache mapping table (DCMT) for storing the mapping of logical block addresses to physical block addresses. This dynamic cache mapping table (DCMT) is dynamically adjusted as the access mode changes to improve the cache hit rate and data access efficiency;
[0058] Step 2, Multi-version Concurrency Control (MVCC): When writing data, introduce multi-version concurrency control (MVCC). Multi-version concurrency control (MVCC) allows the old version of data to be retained while writing, responding to concurrent access requests and avoiding the "write-through" phenomenon;
[0059] Step 3, Delayed Write Correction (DCW): The Delayed Write Correction (DWC) algorithm is used to avoid data inconsistency caused by frequent write operations. The data is written from the cache to the storage medium by delaying, and the data is corrected and verified during the delay period to ensure the consistency and correctness of the data when it is written to the medium.
[0060] Step 4, cache elimination and consistency check: Use the least recently used strategy (LRU) to eliminate cache and perform consistency check before elimination to ensure that all data has been correctly written to the storage medium when it is removed from the cache.
[0061] It should be noted that in step 1, when the system is initialized, a dynamic cache mapping table (DCMT) is created to store the mapping of logical block addresses to physical block addresses. The specific implementation process of this dynamic cache mapping table (DCMT) dynamically adjusting as the access mode changes is as follows:
[0062] Step 11, feature collection and analysis: When the system is initialized, start collecting the access frequency of the logical block address (LBA) (recording the number of accesses and time intervals for each LBA), historical access patterns (identifying and recording the access patterns of each LBA in different time periods, including peak time and access periodicity), and data types (recording the data type stored in the LBA, such as random data or sequential data, and its impact on system performance), analyze these collected features, and generate an initial feature matrix (this matrix is used to calculate the dynamic adjustment factor to help determine which LBAs should be stored in the cache first);
[0063] Step 12, creating an initial mapping table: based on the initial feature matrix, a mapping table from logical block address (LBA) to physical block address (PBA) is preliminarily generated. The mapping table is generated using a high access frequency priority rule. Logical block addresses (LBA) with high access frequency are preferentially mapped to faster physical storage units. According to the data type of the logical block address (LBA), a suitable data storage unit is matched (for sequential data, a unit with higher sequential read performance is selected), and an initial dynamic adjustment factor is generated for each logical block address (LBA);
[0064] Step 13, dynamic adjustment mechanism: When the system is running, the access mode, system load and feature changes of each logical block address (LBA) are continuously monitored (the monitored data will be used to update the feature matrix and system load matrix in real time), and the adjustment factor of each logical block address (LBA) is dynamically adjusted based on the latest monitoring data. The update frequency of the adjustment factor is set according to the fluctuation of the system load and access frequency (for example, adjustment is performed at regular intervals or when a certain threshold is reached). According to the updated adjustment factor, the mapping of the logical block address (LBA) to the physical block address (PBA) is recalculated (if the adjustment factor of a logical block address (LBA) increases significantly, it will be mapped to a storage unit with higher performance to improve the overall system performance);
[0065] Step 14, data migration and synchronization: When updating the mapping table, the logical block address (LBA) changes, and data migration is performed. The data is migrated from the old address to the new address, and the data integrity and consistency of the migration process are guaranteed. After the mapping table is updated, the updated mapping relationship is synchronized to the cache and storage controller (to ensure that all parts of the system are consistent with the latest mapping relationship);
[0066] Step 15, system verification and optimization: Regularly perform consistency checks on the dynamic cache mapping table to ensure the correctness of the mapping relationship, especially in scenarios with high concurrent access and frequent write operations, to prevent the "write-through" phenomenon. Based on feedback from system operation, continuously optimize the calculation rules of the adjustment factor and the update mechanism of the mapping table to further improve the system's response speed and reliability.
[0067] By introducing a dynamic adjustment mechanism of the dynamic cache mapping table (DCMT), this method realizes efficient data management and mapping optimization in semiconductor storage systems. The process from step 11 to step 15 ensures that the system can generate an optimized initial mapping table according to the access frequency, historical pattern and data type of the logical block address (LBA) during initialization, and responds to changes in system load and access pattern in real time through continuous monitoring and dynamic adjustment mechanisms. Data migration and synchronization ensure the consistency of data after the mapping relationship is updated, while system verification and optimization prevent data inconsistency problems such as "write-through" through regular inspection and feedback mechanisms. Overall, this dynamic adjustment mechanism improves the system's performance, response speed and data reliability, especially in scenarios with high concurrent access and frequent write operations, effectively solving the problems of data consistency and system stability.
[0068] It should be noted that in step 13, the mapping from the logical block address (LBA) to the physical block address (PBA) is recalculated according to the updated adjustment factor, and the mapping function is:
[0069] ;
[0070] Where: is the logical block address, is the mapping function value of the logical block address, is the physical block address corresponding to the logical block address, is the dynamic adjustment factor, obtained by calculation, Is the current timestamp.
[0071] When system load or access patterns change, The mapping relationship can be adjusted so that frequently accessed LBAs are preferentially mapped to storage units with higher performance, thus improving data access speed. The dynamic nature of the system ensures that it can adapt to changing workloads and reduce data inconsistency issues caused by high concurrent access or frequent write operations. In addition, this mapping mechanism can reallocate storage resources when needed, optimize overall system performance, and maintain data consistency and system stability in complex storage environments, especially in high-concurrency scenarios, significantly enhancing the reliability and response efficiency of the storage system.
[0072] It should be noted that in the mapping function formula in step 13, the three-dimensional tensor representing the composite relationship between the feature matrix, the load matrix, and the priority matrix is multiplied with the feature matrix in the first dimension, and then multiplied with the matrix obtained by the element-by-element operation of the system load matrix exponentially. The obtained matrix is added to the matrix obtained by the element-by-element operation of the mapping priority matrix sine, and the obtained matrix is used as the dynamic adjustment factor. The formula of the dynamic adjustment factor is:
[0073] ;
[0074] Where: is a three-dimensional tensor representing the interaction between the feature matrix, system load matrix, and priority matrix corresponding to the current timestamp. is a feature matrix used to capture the access frequency, historical access pattern, and data type dimension characteristics of each logical block address (LBA). The feature dimension is , the feature matrix dimension is ,in, is the number of LBAs, is the number of feature dimensions, , are the product operations of tensor and matrix in the first and second dimensions respectively. is the system load matrix, which represents the system at timestamp The load condition of , is the mapping priority matrix, which indicates the mapping priority of each LBA, and its dimension is , , , They are the system load adjustment coefficient, mapping priority adjustment coefficient and frequency adjustment coefficient. for The matrix obtained after performing element-by-element operation on index e, for The matrix resulting from the element-by-element operation of sine.
[0075] In the mapping function formula of step 13, dynamic adjustment factors are constructed by using complex tensor operations and element-by-element operations. , this method realizes the refined management of the mapping relationship between logical block address (LBA) and physical block address (PBA). Specifically, the three-dimensional tensor It represents the interaction between the feature matrix, system load matrix and priority matrix. The product in the first dimension and the system load matrix The product of the exponential element-by-element operation ensures that the system can dynamically adjust the mapping relationship according to the current load and LBA characteristics. At the same time, through the mapping priority matrix The sinusoidal element-by-element operation captures the periodic changes in the LBA access pattern, allowing the system to flexibly adjust the mapping relationship under different priority and access frequency conditions. The specific help of this setting is: 1) improving the performance and response speed of the system in high-concurrency scenarios, by adaptively adjusting the mapping relationship to ensure that the LBA with high frequency access is given priority; 2) enhancing data consistency and reliability, the dynamic adjustment factor can balance the system load, avoid excessive concentration of resources, and reduce data inconsistency problems; 3) providing a flexible mapping adjustment mechanism that can be optimized according to real-time load and access characteristics, adapt to changing workloads, and improve the stability and performance of the overall system.
[0076] is a coefficient used to control the degree to which the system load affects the dynamic adjustment factor. It determines how the adjustment factor should respond when the system load is high. The higher the load, The larger the value, the slower the growth of the adjustment factor, which prevents system resources from being overly concentrated on certain LBAs, thereby avoiding system bottlenecks. The value of is usually derived from statistical analysis of the following data sources: (1) Real-time system monitoring data: including CPU utilization, memory usage, I / O operation number, etc., which directly reflects the current load of the system; (2) Historical load analysis: by analyzing the load pattern of historical data, the appropriate load for the system is calculated. (3) Performance test results: Based on system performance testing, determine the optimal value to balance performance and resource utilization.
[0077] It is used to control the impact of mapping priority on the adjustment factor. It determines to what extent the system should consider the priority of LBA when allocating resources. High-priority LBA will get more resource allocation to ensure that these important data blocks can be accessed quickly. The value of can be determined by the following data sources: (1) Data importance index: According to the system's assessment of the importance of data blocks (such as key business data, real-time data, etc.), different priorities are assigned, which in turn affects (2) User settings or policies: System administrators or users can manually adjust the priority of LBAs according to business needs to reflect current business priorities; (3) Access pattern analysis: By analyzing the historical access patterns of LBAs, their importance in future accesses is calculated, so as to set reasonable value.
[0078] It is a coefficient used to control the periodic effect of access frequency on the adjustment factor. It helps capture the periodic characteristics of LBA access in the system and adapt to the access rules of different LBAs. A value of 0 means that the adjustment factor will more sensitively reflect periodic changes in access frequency. The value of is based on the following data sources: (1) Access frequency statistics: By monitoring the access frequency of each LBA in real time, its periodic characteristics are determined, and the appropriate (2) Historical access data: Analyze long-term access records, identify LBA access periodicity patterns, and set them accordingly. ; (3) Specific requirements of usage scenarios: In some specific scenarios (such as real-time data analysis or high-frequency trading systems), The value will be adjusted according to the needs of the scenario to better adapt to high-frequency and periodic access patterns.
[0079] It should be noted that in step 2, the specific implementation process of multi-version concurrency control (MVCC) includes:
[0080] Step 21, data version management: In the storage system, each logical block address (LBA) pair will be written multiple times at different time points. A new data version is generated for each write operation, while the old data version is retained. A unique version number is set for each data version using the version management function to identify the data generation time. The data version of each logical block address (LBA) is recorded in a version chain, which is sorted by timestamp, with the latest version at the end of the chain.
[0081] Step 22, data read operation: When the system needs to read data, the read operation selects a suitable version from the version chain according to the data read function based on the current system time and transaction timestamp to read, so as to ensure that the read operation will not be affected by concurrent write operations;
[0082] Step 23, data writing operation: When performing the data writing operation, the system does not directly overwrite the existing data version, but uses the data writing function to generate a new version of the data and marks it with the current timestamp (so that even in the case of concurrent writing, the old version of the data is still available to ensure data consistency and integrity);
[0083] Step 24, version garbage collection: regular cleaning based on the version recycling function or intelligent cleaning based on the access pattern to release the storage space occupied by the version garbage.
[0084] By introducing multi-version concurrency control (MVCC), this method effectively solves the problem of data inconsistency caused by concurrent write operations in semiconductor storage systems. Specifically, data versioning management ensures that write operations for each logical block address (LBA) at different time points will generate new data versions, and retain the old versions to maintain data history. Data read operations can select the appropriate version for reading without being affected by concurrent writes to ensure data consistency. Data write operations avoid directly overwriting old data by generating new versions to ensure data integrity. Version garbage collection optimizes storage space utilization by cleaning up old versions that are no longer needed. Overall, these measures significantly improve system stability, data reliability, and storage efficiency, especially in high-concurrency environments.
[0085] It should be noted that in step 21 of the specific implementation process of step 2, the version management function is:
[0086] ;
[0087] Where: is the data version index, The timestamp for generating the data version. For When writing data to the logical block address (LBA), For The version management function value corresponding to the data written to the logical block address (LBA) when the
[0088] It should be noted that in step 22 of the specific implementation process of step 2, the formula of the data reading function is:
[0089] ;
[0090] Where: is the timestamp of the current transaction, For all less than or equal to Select the maximum timestamp Data version To read, For the current time point The latest version of the data available.
[0091] It should be noted that in step 23 of the specific implementation process of step 2, the formula of the data writing function is:
[0092] ;
[0093] Where: The time when the current new data version is generated. The generated new data version contains the new data and timestamp .
[0094] It should be noted that in step 24 of the specific implementation process of step 2, the version recovery function is:
[0095] ;
[0096] Where: Remember the recycling operation for the version to delete the old version data that is no longer needed. The time threshold is set, and versions below this timestamp are recycled.
[0097] It should be noted that in step 3, the specific implementation process of the delayed write correction algorithm is as follows:
[0098] Step 31, cache data marking and delay queue management: when data is written into the cache, each data block is marked as uncommitted, indicating that the data has not been written into the physical storage medium. The data is put into the delay write queue. After the data enters the delay write queue, the queue is sorted according to the time sequence of the write request, and the initial write time of each data block is recorded;
[0099] Step 32, delayed writing and data verification: after the data enters the delay queue, the time when the data is written into the physical storage medium is controlled according to the preset delay time. If the data does not change during the delay period, it is written according to the planned time. If the data changes, the delay time is recalculated and verified. During the delay period, the system verifies the data in the delay queue to detect whether there is a data conflict or inconsistency. If a conflict or data error is found, the data is corrected to ensure that the data finally written is correct.
[0100] Step 33, submission and consistency check: When the data is finally written to the physical storage medium, the system writes the data in the delay queue to the physical storage medium. After writing, the data status changes from unsubmitted to submitted. After the data is written to the medium, the system performs a consistency check again to ensure that no errors or loss occur during the writing process. If the check passes, the data status changes to submitted.
[0101] By introducing a delayed write correction algorithm, the system can effectively solve the problem of data inconsistency caused by frequent write operations. Specifically, cache data marking and delayed queue management ensure that the data status is clearly marked and managed in order before the data is officially written to avoid confusion. The delayed write and data verification mechanism ensures that data conflicts or errors can be detected and corrected during the writing process through flexible control and verification of the delay time, preventing inconsistent data from being written. The final submission and consistency check further ensure the integrity and correctness of the written data. Overall, these measures improve the reliability of the system and reduce errors caused by write operations, especially in scenarios with high concurrency and frequent writes, significantly improving the stability and consistency of the data.
[0102] It should be noted that in step 4, the specific implementation process of using the least recently used strategy (LRU) for cache elimination is:
[0103] Step 41, cache access record and sorting: the system maintains an access record for each logical block address (LBA) in the cache. Each time a data block is accessed, the access timestamp of the data block is updated. All data blocks in the cache are sorted according to the access timestamp, with the most recently accessed blocks placed at the front and the least recently accessed blocks placed at the end. In this way, the data blocks in the cache are arranged in order of access frequency and time.
[0104] Step 42, cache elimination strategy execution: When the cache space is insufficient and space needs to be freed, the system selects the data block that has not been accessed for the longest time for elimination through sorting, that is, selects the last data block in the sorted list for removal, and the selected data block is removed from the cache. If the content of the data block has not been written to the storage medium, it will be written to the storage medium before removal to ensure that the data is not lost.
[0105] Step 43, data consistency check: when executing cache elimination, the system performs data consistency check to ensure that the removed data block has been correctly written to the storage medium. If the consistency check fails, the system will prevent the elimination of the data block until the problem is resolved.
[0106] Step 44, cache update and synchronization: After the data block in the cache is removed, the space in the cache is released, and the system can continue to cache new data blocks. At this time, the cache records and sorting lists will also be updated accordingly, and the latest cache status will be synchronized to other modules of the system to ensure that the entire system synchronizes the cache status.
[0107] The LRU strategy is used for cache elimination, which effectively solves the resource management problem when the cache space is limited. By giving priority to eliminating the longest-unused data blocks, the LRU strategy ensures that the frequently accessed data blocks can be retained in the cache as much as possible, thereby improving the system's response speed. At the same time, the data consistency check before elimination ensures that important data will not be lost when the data blocks are removed, maintaining the stability of the system and data integrity. Overall, the implementation of the LRU strategy improves the system's cache management efficiency, reduces unnecessary I / O operations, and enhances the performance and reliability of the storage system under high load conditions.
[0108] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0109] Finally: The above description is only an implementation scheme of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A data management method for semiconductor storage, characterized in that: The steps include: Step 1, initialize the dynamic cache mapping table: When the system is initialized, create a dynamic cache mapping table for storing the mapping of logical block addresses to physical block addresses. This dynamic cache mapping table obtains dynamic adjustment factors based on the latest monitoring data of each physical block address as the access mode changes, so as to recalculate the mapping of logical block addresses to physical block addresses. The mapping function is: M(LBA)=PBA(LBA)+Δ(t); Where: LBA is the logical block address, M(LBA) is the mapping function value of the logical block address, PBA(LBA) is the physical block address corresponding to the logical block address, Δ(t) is the dynamic adjustment factor obtained by calculation, and t is the current timestamp; Step 2: Multi-version concurrency control: When writing data, introduce multi-version concurrency control to allow the old version of data to be retained while writing; Step 3, delayed write correction: Use a correction algorithm to write data from the cache to the storage medium, and correct and verify the data during the delay period; the specific implementation process of the correction algorithm is as follows: Step 31, cache data marking and delay queue management: when data is written into the cache, each data block is marked as uncommitted. After the data enters the delay write queue, the queue is sorted according to the time sequence of the write request, and the initial write time of each data block is recorded; Step 32, delayed writing and data verification: after the data enters the delay queue, the time when the data is written into the physical storage medium is controlled according to the preset delay time. If the data does not change during the delay period, it is written according to the planned time. If the data changes, the delay time is recalculated and verified. During the delay period, the system verifies the data in the delay queue to detect whether there is a data conflict or inconsistency. If a conflict or data error is found, the data is corrected; Step 33, commit and consistency check: When the data is finally written to the physical storage medium, the system writes the data in the delay queue to the physical storage medium. After writing, the data status changes from uncommitted to committed. After the data is written to the medium, the system performs a consistency check again. If the check passes, the data status changes to committed. Step 4: Cache elimination and consistency check: Use the least recently used strategy to eliminate cache and perform consistency check before elimination.
2. A data management method for semiconductor storage according to claim 1, characterized in that: In step 1, the specific implementation process of dynamically adjusting the dynamic cache mapping table as the access mode changes is as follows: Step 11, feature collection and analysis: When the system is initialized, the access frequency, historical access pattern, and data type of the logical block address are collected, and the collected features are analyzed to generate an initial feature matrix; Step 12, creating an initial mapping table: based on the initial feature matrix, a mapping table from logical block addresses to physical block addresses is preliminarily generated. The mapping table is generated using a high access frequency priority rule, and logical block addresses with high access frequency are preferentially mapped to faster physical storage units. According to the data type of the logical block address, a suitable data storage unit is matched, and an initial dynamic adjustment factor is generated for each logical block address; Step 13, dynamic adjustment mechanism: when the system is running, the access mode, system load and characteristic changes of each logical block address are continuously monitored, and the adjustment factor of each logical block address is dynamically adjusted based on the latest monitoring data. The update frequency of the adjustment factor is set according to the fluctuation of the system load and the access frequency. According to the updated adjustment factor, the mapping of the logical block address to the physical block address is recalculated; Step 14, data migration and synchronization: when updating the mapping table, the logical block address changes, and data migration is performed, the data is migrated from the old address to the new address, and the data integrity and consistency of the migration process are ensured. After the mapping table is updated, the updated mapping relationship is synchronized to the cache and storage controller; Step 15, system verification and optimization: Regularly perform consistency verification on the dynamic cache mapping table. In scenarios with high concurrent access and frequent write operations exceeding the preset threshold, based on feedback from system operation, continuously optimize the calculation rules of the adjustment factor and the update mechanism of the mapping table.
3. A data management method for semiconductor storage according to claim 2, characterized in that: In the mapping function formula in step 13, the three-dimensional tensor is multiplied by the feature matrix in the first dimension to obtain a new matrix, and then the exponential decay of each element in the load matrix is calculated to obtain the load index matrix, and the load index matrix and the new matrix are multiplied in the second dimension to obtain the result matrix, and then the priority matrix is element-by-element sine operation is performed to obtain the priority adjustment matrix, and finally the sum of the result matrix and the priority adjustment matrix is used as the dynamic adjustment factor. The formula of the dynamic adjustment factor is: Where: is a three-dimensional tensor representing the interaction between the feature matrix corresponding to the current timestamp, the system load matrix, and the priority matrix. F(t) is the feature matrix, which is used to capture the access frequency, historical access pattern, and data type dimension characteristics of each logical block address. ×1 and ×2 are the product operations of the tensor and the matrix on the first and second dimensions, respectively. W(t) is the system load matrix, P(t) is the mapping priority matrix, α, β, and γ are the system load adjustment coefficient, mapping priority adjustment coefficient, and frequency adjustment coefficient, respectively. exp(-α·W(t)) is the matrix obtained by performing the e-exponential element-by-element operation on W(t), and sin(γ·P(t)) is the matrix obtained by performing the sine element-by-element operation on P(t).
4. A data management method for semiconductor storage according to claim 1, characterized in that: In step 2, the specific implementation process of multi-version concurrency control includes: Step 21, data version management: perform multiple write operations on each logical block address in the storage system at different time points, generate a new data version for each write operation, and retain the old data version. Use the version management function to set a unique version number for each data version to identify the generation time of the data. The data version of each logical block address is recorded in a version chain, which is sorted by timestamp, and the latest version is at the end of the chain; Step 22, data reading operation: when the system needs to read data, the reading operation selects a suitable version from the version chain according to the data reading function based on the current system time and transaction timestamp for reading; Step 23, data writing operation: when performing the data writing operation, the system does not directly overwrite the existing data version, but uses the data writing function to generate a new version of the data and marks it with the current timestamp; Step 24, version garbage collection: regular cleaning based on the version recycling function or intelligent cleaning based on the access pattern to release the storage space occupied by the version garbage.
5. A data management method for semiconductor storage according to claim 4, characterized in that: In step 21 of the specific implementation process of step 2, the version management function is: V(LBA,t i )={(data,t i )|data}; Where: i is the data version index, t i The timestamp of the data version. i The data written to the logical block address, V(LBA, t i ) is at t i The version management function value corresponding to the data written to the logical block address when 6. A data management method for semiconductor storage according to claim 5, characterized in that: In step 22 of the specific implementation process of step 2, the formula of the data reading function is: Where: t cur is the timestamp of the current transaction, For all values less than or equal to t cur Among the timestamps, select the maximum t i Data version V(LBA, t i ) to read, R(LBA, t cur ) is the current time point t cur The latest version of the data available.
7. A data management method for semiconductor storage according to claim 5, characterized in that: In step 23 of the specific implementation process of step 2, the formula of the data writing function is: R(LBA,t new ,data)=(data,t new ); Where: t new The time when the current new data version is generated, R(LBA, t new , data) is the generated new data version, including the new data data and timestamp t new .
8. A data management method for semiconductor storage according to claim 5, characterized in that: In step 24 of the specific implementation process of step 2, the version recovery function is: GC(LBA)={Remove(V(LBA,t i )),t i <t thr }; Where: GC(LBA) is the version-remembering recycling operation, which is used to delete old version data that is no longer needed, t thr The time threshold is set, and versions below this timestamp are recycled.
Citation Information
Patent Citations
Memory consistency protection in a multiprocessor computing system
CN101542441A
Systems and methods for write and flush support in hybrid memory
CN108874701A
Apparatus and method for controlling map data in a memory system
CN113360083A
