Methods, apparatus, devices, storage media, and program products for adjusting storage location.

Abstract

This application discloses a method, apparatus, device, storage medium, and program product for adjusting storage locations, relating to the field of storage technology. The method includes: allocating a group of data blocks to different storage spaces pre-divided in a cache according to a preset strategy; the different storage spaces include: a first storage space allowing execution of a preset log mechanism and a second storage space allowing synchronous updates with permanent storage; receiving access operations on the data blocks allocated to the different storage spaces at preset time intervals; and adjusting the storage location of the data blocks allocated to the different storage spaces in the different storage spaces based on the first access frequency corresponding to the access operations. Therefore, this method can solve the problem of performance bottlenecks and resource waste caused by the write-before-log mechanism used in storage systems, which records all operations on data in the cache in chronological order without distinguishing the access frequency of different types of data, thus overcoming performance bottlenecks and saving storage system resources.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a method, apparatus, device, storage medium, and program product for adjusting storage location. Background Technology

[0002] Storage systems in related technologies, such as database management systems, distributed file systems, and mission-critical platforms, commonly employ Write-Ahead Logging (WAL) mechanisms to ensure data consistency and integrity in the event of system failures and to support rapid recovery. The basic principle of this mechanism is to write change records to a log file before actually writing data, allowing data recovery through log replay in case of anomalies. However, mainstream WAL mechanisms typically record all operations in chronological order, failing to differentiate between the access frequencies of different data types. This leads to significant performance bottlenecks and resource waste, especially when handling frequently accessed hot data.

[0003] In related technologies, the write-before-log mechanism used by storage systems records all operations on cached data in chronological order, without distinguishing the access frequency of different types of data, resulting in performance bottlenecks and resource waste, which have not yet been effectively resolved. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and program product for adjusting storage location, to at least solve the problem that the write-before-log mechanism used in related technologies records all operations on data in the cache in chronological order, without distinguishing the access frequency of different types of data, resulting in performance bottlenecks and resource waste.

[0005] This application provides a method for adjusting storage location, comprising: allocating a group of data blocks to different storage spaces pre-divided in a cache according to a preset strategy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronous updates with permanent storage, the preset strategy being used to instruct data blocks within the same access frequency range to be allocated to the same storage space; receiving access operations on the data blocks allocated in the different storage spaces at preset time intervals, and adjusting the storage location of the data blocks allocated in the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation.

[0006] This application also provides a storage location adjustment device, comprising: an allocation module, configured to allocate a group of data blocks to different storage spaces pre-divided in a cache according to a preset strategy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronous updates with permanent storage, the preset strategy being used to instruct data blocks within the same access frequency range to be allocated to the same storage space; and an adjustment module, configured to receive access operations on the data blocks allocated in the different storage spaces at preset time intervals, and adjust the storage location of the data blocks allocated in the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described methods for adjusting the storage location when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described storage location adjustment methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described storage location adjustment methods.

[0010] This application allocates a set of data blocks to different storage spaces pre-defined in the cache according to a preset strategy. These different storage spaces include a first storage space that allows execution of a preset logging mechanism and a second storage space that allows synchronous updates with permanent storage. The preset strategy instructs data blocks with the same access frequency to be allocated to the same storage space. Access operations on the data blocks allocated to the different storage spaces are received at preset time intervals. The storage location of the data blocks allocated to the different storage spaces is adjusted based on the first access frequency corresponding to the access operations. Therefore, this solution addresses the technical problem in related technologies where the write-before-log mechanism records all operations on data in the cache in chronological order, failing to differentiate access frequencies for different data types, leading to performance bottlenecks and resource waste. This solution achieves the technical effect of overcoming performance bottlenecks and saving storage system resources. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a hardware structure block diagram of a computer terminal for a method of adjusting storage location according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of a method for adjusting the storage location according to an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the storage space of the storage location adjustment method according to an embodiment of this application;

[0015] Figure 4 This is another flowchart of a method for adjusting the storage location according to an embodiment of this application;

[0016] Figure 5 This is a structural block diagram of a storage location adjustment device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The specific application environment architecture or specific hardware architecture on which the method for adjusting the storage location depends is described here.

[0021] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of adjusting storage location according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0022] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the storage location adjustment method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0023] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0024] Figure 2 This is a flowchart of a method for adjusting the storage location according to an embodiment of this application. Figure 2 As shown, the process includes the following steps:

[0025] Step S202: According to a preset strategy, a group of data blocks are allocated to different storage spaces pre-divided in the cache. The different storage spaces include: a first storage space that allows the execution of a preset log mechanism and a second storage space that allows synchronous updates with permanent storage. The preset strategy is used to indicate that data blocks with the same access frequency range are allocated to the same storage space.

[0026] Step S204: Receive access operations to data blocks allocated in different storage spaces according to a preset time interval, and adjust the storage location of the data blocks allocated in different storage spaces in different storage spaces according to the first access frequency corresponding to the access operation.

[0027] Through the above steps, a group of data blocks are allocated to different storage spaces pre-divided in the cache according to a preset strategy. These different storage spaces include a first storage space that allows execution of a preset logging mechanism and a second storage space that allows synchronous updates with permanent storage. The preset strategy instructs data blocks with the same access frequency to be allocated to the same storage space. Access operations on the data blocks allocated to the different storage spaces are received at preset time intervals. The storage location of the data blocks allocated to the different storage spaces is adjusted based on the first access frequency corresponding to the access operations. Therefore, this solves the technical problem in related technologies where the write-before-log mechanism used in storage systems records all operations on data in the cache in chronological order, without distinguishing the access frequency of different data types, leading to performance bottlenecks and resource waste. This achieves the technical effect of overcoming performance bottlenecks and saving storage system resources.

[0028] The embodiments of this application provide a method for adjusting the storage location. The method is described in detail below in conjunction with the execution flow of the method for adjusting the storage location.

[0029] In one exemplary embodiment, allocating a set of data blocks to different storage spaces pre-divided in the cache according to a preset strategy includes: dividing the set of data blocks into multiple data block groups with different priorities according to the preset strategy; and allocating the multiple data block groups to the different storage spaces according to the priorities corresponding to the multiple data block groups respectively.

[0030] This application optimizes data storage, improves system performance, and enhances stability and fault tolerance, which has significant practical value for large-scale distributed storage and database systems.

[0031] Optionally, the set of data blocks is divided into multiple data block groups with different priorities according to the preset strategy, including: obtaining metadata information of each data block in the set of data blocks; dividing the multiple data block groups with different priorities based on the metadata information according to the preset strategy, wherein each data block group in the multiple data block groups includes: data blocks in the set of data blocks that are in the same access frequency range.

[0032] Metadata information includes: the identifier, initial access count, access time, and category of each data block in a set of data blocks. The access frequency of each data block can then be determined using the initial access count from the metadata information. Based on the relationship between the access frequency and several preset frequency thresholds, or the relationship between the access frequency and several preset access frequency ranges, each data block in a set of data blocks can be divided into multiple data block groups. For example, the multiple data block groups could be: a high-frequency cache data block group, where data blocks with an access frequency greater than frequency threshold A1 can be assigned to the high-frequency cache data block group; a medium-frequency cache data block group, where data blocks with an access frequency between frequency threshold A1 and frequency threshold B1 can be assigned to the medium-frequency cache data block group, etc.

[0033] Optionally, allocating the plurality of data block groups to different storage spaces according to their respective priorities includes: determining the correspondence between the priorities and the different storage spaces; determining a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space through the correspondence, wherein the plurality of data block groups includes: the first data block group and the second data block group; determining a first number of data blocks that can be recorded in the first storage space and a second number of data blocks that can be recorded in the second storage space; recording data blocks in the first data block group into the first storage space according to the first number, and recording data blocks in the second data block group into the second storage space according to the second number, so as to allocate the plurality of data block groups to the different storage spaces.

[0034] For example, the first data block group can be a high-frequency cached data block group, stored in a first storage space that can execute a preset logging mechanism (such as a pre-write logging mechanism), and all read and write operations on the cached data blocks in the first storage space are pre-written to the log corresponding to the preset logging mechanism; the second data block group can be a medium-frequency cached data block group, stored in a second storage space that is synchronously updated with permanent storage, and all read and write operations on the cached data blocks in the second storage space are synchronously updated with permanent storage.

[0035] The strategy of dynamically allocating data blocks to different storage spaces according to priority, as proposed in this application, can not only significantly improve system performance, but also optimize resource utilization and enhance the security and reliability of data management. It has important technical significance and practical value for data storage management in distributed database systems, data-intensive applications, and high-load environments.

[0036] In one exemplary embodiment, adjusting the storage location of data blocks allocated in different storage spaces according to a first access frequency corresponding to the access operation includes: obtaining a second access frequency corresponding to w data blocks, wherein the w data blocks are data blocks allocated in each of the different storage spaces, and w is a positive integer; sorting the w data blocks according to the second access frequency; and adjusting the storage location of the data blocks allocated in the different storage spaces according to the sorted w data blocks.

[0037] The second access frequency is the frequency at which each of the w data blocks is accessed within a preset time interval.

[0038] It should be noted that the access frequency in this application refers to the access frequency.

[0039] Furthermore, sorting the w data blocks by the second access frequency includes: obtaining the target time corresponding to each of the w data blocks, wherein the target time is used to indicate the time interval between the last access time of each of the w data blocks and the current time; and sorting the w data blocks by the target time and the second access frequency.

[0040] Specifically, sorting the w data blocks by the target time and the second access frequency includes: normalizing the target time and the second access frequency respectively; performing a weighted summation of the normalized target time and the normalized second access frequency to obtain a popularity value corresponding to each of the w data blocks; and sorting the w data blocks by the popularity value.

[0041] In other words, the heat value can be calculated using the following formula:

[0042] .

[0043] in, This represents the heat value corresponding to each of the w data blocks. This represents the normalization function, and you can choose functions such as min-max normalization. This represents the second access frequency of each of the w data blocks. Indicates the target time. and Indicates the weight.

[0044] This application demonstrates how a heat value is generated by normalizing and weighting the target time and access frequency, and then sorting the data blocks. This significantly improves the intelligence level of cache management, optimizes data access performance, and adapts to the needs of different business scenarios. It reflects the importance of dynamic resource management and data heat perception, and has significant technical implications for building a high-performance, highly adaptive distributed storage system.

[0045] In one exemplary embodiment, adjusting the storage location of data blocks allocated in different storage spaces using sorted w data blocks includes: determining the adjustment order corresponding to the different storage spaces according to priority, thereby determining a third storage space and / or a fourth storage space in the different storage spaces, wherein the priority is the priority corresponding to a plurality of data block groups allocated to the different storage spaces, the plurality of data block groups includes the plurality of data block groups, the third storage space is the storage space whose adjustment order precedes each storage space, and the fourth storage space is the storage space whose adjustment order follows each storage space; adjusting the target data block group from the sorted w data blocks to the third storage space and / or the fourth storage space to adjust the storage location of the data blocks allocated in the different storage spaces.

[0046] For example, if the high-frequency cache data block group has a higher priority than the medium-frequency data block group, then the storage space used to store the high-frequency cache data block group will be adjusted before the storage space used to store the medium-frequency data block group.

[0047] Further, the target data block group from the sorted w data blocks is adjusted to the third storage space and / or the fourth storage space to adjust the storage location of the data blocks allocated in the different storage spaces, including: exchanging the third data block group in each storage space with the fourth data block group in the third storage space; and / or exchanging the fifth data block group in each storage space with the sixth data block group in the fourth storage space; wherein, the target data block group includes: the third data block group and / or the fifth data block group; the third data block group includes: the first preset proportion of data blocks ranked first among the sorted w data blocks in each storage space, the fifth data block group includes: the second preset proportion of data blocks ranked last among the sorted w data blocks in each storage space; the fourth data block group includes: the first preset proportion of data blocks ranked last among the sorted w data blocks in the third storage space, and the sixth data block group includes: the second preset proportion of data blocks ranked first among the sorted w data blocks in the fourth storage space.

[0048] For example, given high-frequency, mid-frequency, and low-frequency storage spaces, the last 30% of the sorted w data blocks in the high-frequency storage space are swapped with the first 30% of the sorted w data blocks in the mid-frequency storage space, and vice versa. The last 30% of the sorted w data blocks in the mid-frequency storage space are swapped with the first 30% of the sorted w data blocks in the low-frequency storage space. For the last 30% of the sorted w data blocks in the low-frequency storage space, they can be swapped with data blocks not stored in the pre-allocated storage spaces. For instance, if a group of data blocks allocated to the high-frequency cache block group is not entirely stored in the high-frequency storage space but instead stored on the data disk, then the last 30% of the sorted w data blocks in the low-frequency storage space can be swapped with data blocks on the data disk.

[0049] For example, the data blocks in the first storage space can be sorted after calculating their popularity value, while other storage spaces, including the second storage space, can be sorted based solely on the second access frequency of each of the w data blocks. Then, the positions of the data blocks in different storage spaces are swapped according to the storage space adjustment order, a first preset ratio, and a second preset ratio.

[0050] In an exemplary embodiment, after adjusting the storage location of the data blocks allocated in the different storage spaces according to the first access frequency corresponding to the access operation, the method further includes: flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage; creating a second log file corresponding to the preset log mechanism, wherein the second log file is used to pre-write the read and write operations corresponding to the data blocks stored in the fifth storage space in the next time interval, and the fifth storage space is the first storage space after the storage location of the stored data blocks has been adjusted.

[0051] Furthermore, before flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage, the method further includes: in the event of an abnormal event, restoring the data blocks in the first storage space that have not been written to the permanent storage to the last consistent state through the replay mechanism corresponding to the first log file.

[0052] This application, through the WAL mechanism's periodic data flushing, log file reconstruction, and data recovery strategies under abnormal events, can effectively ensure data persistence and consistency, while optimizing system performance and resource utilization, improving fault recovery efficiency, simplifying operation and maintenance, and enhancing system adaptability and stability. It has significant technical effects on the high efficiency and high reliability of distributed storage systems.

[0053] To better understand the process of the above-mentioned storage location adjustment method, the implementation flow of the above-mentioned storage location adjustment method will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0054] The optional embodiments of this application are mainly applied to the data caching problem in storage systems. By identifying hot data and selectively caching data, combined with a pre-write logging mechanism, the performance of the storage system can be significantly improved.

[0055] Storage systems in related technologies, such as database management systems, distributed file systems, and mission-critical platforms, commonly employ a write-before-log (WAL) mechanism to ensure data consistency and integrity in the event of system failure and to support rapid recovery. The basic principle of this mechanism is to write change records to a log file before actually writing data, allowing data recovery through log replay in case of anomalies. However, current mainstream WAL mechanisms typically record all operations in chronological order, without distinguishing the access frequency of different data types. This leads to significant performance bottlenecks and resource waste, especially when handling frequently accessed hot data.

[0056] Specifically, firstly, hot data blocks, due to frequent access and modification, generate a large number of repetitive log entries, leading to severe write amplification and a rapid increase in storage resource consumption. This redundant writing not only wastes disk input / output (I / O) but also increases the system load and reduces overall throughput performance. Secondly, traditional caching mechanisms typically do not dynamically optimize for the evolution of access patterns, failing to accurately identify and prioritize the retention of hot data, resulting in a low cache hit rate, reduced read performance, and indirectly increased burden on the log system. Furthermore, during system failure recovery, the WAL mechanism often requires sequential replay of all log records to restore a consistent state, even if most data has already been persisted and unchanged. This "one-size-fits-all" replay strategy is time-consuming and inefficient, failing to meet modern demands for high availability and rapid recovery.

[0057] Therefore, the existing WAL mechanism has room for improvement in handling hot data and improving fault recovery efficiency, and there is an urgent need to introduce a more adaptive log and cache co-optimization scheme.

[0058] To address the aforementioned shortcomings, this application proposes an optional embodiment of a data caching method based on hot data and pre-write logs in a distributed storage system (equivalent to the storage location adjustment method in the above embodiments). The method includes: setting initial configuration information for data blocks and dividing them into 2-4 priority levels. Then, upper-layer business processes are run to collect the usage frequency of each data block. Based on the data usage frequency, different iteration strategies are adopted to adjust the data cache. The cached data is read and written using pre-write logs. After the next iteration, the data in the pre-write logs is flushed based on the newly identified hot data. If data loss occurs during the process, data recovery is performed based on the pre-write logs, ensuring data security and read / write efficiency.

[0059] Specifically, the basic idea of ​​the optional embodiments of this application is as follows:

[0060] 1) During the initialization phase, set the initial configuration information and priority division of the data blocks.

[0061] When the system starts up or a new data block is written, basic configuration information is set for each data block in a set of data blocks, such as data block identification (ID), initial access count, access timestamp, etc. Then, all data blocks are divided into 2 to 4 different priority levels according to default rules (such as preset access frequency, data type, or application tags). For example, they can be divided into categories such as "high-frequency access," "medium-frequency access," "low-frequency access," and "initially unknown," for subsequent dynamic adjustment.

[0062] 2) Run the business and dynamically collect the access frequency of data blocks.

[0063] The system begins normal operation, executing upper-layer business logic (such as database operations or file read / write). During this process, access information for each data block is collected in real-time or periodically, including read counts, write frequency, and write time. This data is used to assess the "hotness" of the current data block, providing a basis for iteratively adjusting the strategy.

[0064] 3) Iterate based on access frequency.

[0065] Based on the access frequency of the collected data blocks, the priority of each data block is dynamically adjusted using a specified iterative strategy. Data blocks accessed frequently are given higher priority and assigned higher cache weights; while low-frequency data blocks may be downgraded or even removed from the cache, thereby improving the overall cache hit rate.

[0066] 4) Combine the pre-write logging mechanism to manage the reading and writing of cached data.

[0067] The system manages selected cached data blocks using a Write-Ahead Log (WAL) mechanism. All read and write operations on cached data are first written to the WAL file to ensure traceability and recoverability. This not only enhances system fault tolerance but also avoids the frequent I / O overhead caused by direct disk writes.

[0068] 5) Iterative update and cache flushing strategy.

[0069] At the start of the next iteration cycle, the system re-evaluates hot data based on the new data access patterns. Changes to data blocks that are no longer considered hot are flushed to persistent storage (such as disk or object storage) and removed from the cache, thus maintaining efficient use of cache space. Simultaneously, currently hot data is prioritized for retention in the cache.

[0070] 6) Fault recovery mechanism, using pre-write logs to ensure data security.

[0071] If a system crash, power outage, or other anomaly occurs during the above process, the system can use the WAL file replay mechanism to restore data that has not been written to persistent storage to a last consistent state. This mechanism ensures that recovery can be completed through logs even if cached data has not been flushed, significantly improving the system's fault tolerance and data security.

[0072] Furthermore, based on the above basic ideas, the implementation schemes of the optional embodiments of this application are as follows: Figure 3 As shown, it includes:

[0073] Step S31: Initialize the data space and set the initial configuration information and priority division of the data blocks.

[0074] In a distributed storage system, metadata information for data blocks is configured, recording the block ID, initial access count, access time, and category. Data blocks are categorized into four types: high-frequency access blocks, medium-frequency access blocks, low-frequency access blocks, and initial data blocks. The cache space for high-frequency cached data is set as R1, recording L1 data blocks. The cache space for medium-frequency cached data is set as R2, recording L2 data blocks. Figure 4 As shown.

[0075] Step S32: Run the business and dynamically collect data block access frequency and time.

[0076] After running the upper-layer business for a period of time, record the data access frequency. Based on the access frequency, place the first L1+L2 data blocks into the R1 and R2 spaces respectively.

[0077] Step S33: Exchange data locations based on access frequency.

[0078] A fixed time interval T is set (equivalent to the preset time interval in the above embodiment). After time T has elapsed, the access frequency of the R1 space data block and the time interval between the last access time and the current time point are obtained. The data blocks are then sorted using a normalization method. The popularity value used for sorting is calculated as follows:

[0079] .

[0080] in, =0.7, =0.3 ( and (Parameters can be adjusted according to actual conditions). This represents the normalization function, specifically the min-max normalization:

[0081] .

[0082] Take the last 30% of the sorted data in space R1 and swap it with the first 30% of the data in space R2, sorted by access frequency. Simultaneously, swap the last 30% of the data in space R2 with the frequently accessed data on the data disk.

[0083] Step S34: Combine the pre-write log mechanism to manage the reading and writing of cached data.

[0084] During time interval T, data in the R1 cache is first written to the WAL file. Data written to the R2 space simultaneously updates memory and disk. This ensures the operation is traceable and recoverable, while also ensuring that the amount of data processed is not excessive.

[0085] Step S35: Iterative update and cache flushing strategy.

[0086] At the start of the next iteration cycle, after the position swap in step S33 is completed, the data in the WAL log is flushed to the disk, and a new WAL record is created to record the new round of data operations.

[0087] Step S36: Implement the fault recovery mechanism and use the pre-write log to ensure data security.

[0088] If the system crashes due to incomplete data flushing, the next time the data storage is restarted, the WAL data will be flushed first to update the data to the latest version, and then the iteration will start again from step S32.

[0089] Optionally, the data space can be refined to correspond to multiple WALs to improve read and write efficiency; normalization. The function can be other normalization functions, such as Z-Score normalization.

[0090] In summary, the optional embodiments of this application first set the initial configuration information for data blocks, dividing them into 2-4 priority levels. Then, the upper-layer business logic is run to collect the usage frequency of each data block. Based on the data usage frequency, an iterative strategy is adopted to adjust the data cache. The cached data is read and written using a pre-write log. After the next iteration, the data in the pre-write log is flushed down based on the newly identified hot data. If data loss occurs during the process, data recovery is performed based on the pre-write log, ensuring data security and read / write efficiency.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0092] This embodiment also provides a storage location adjustment device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0093] Figure 5 This is a structural block diagram of a storage location adjustment device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0094] The allocation module 52 is used to allocate a group of data blocks to different storage spaces pre-divided in the cache according to a preset strategy. The different storage spaces include: a first storage space that allows the execution of a preset log mechanism and a second storage space that allows synchronous updates with permanent storage. The preset strategy is used to indicate that data blocks with the same access frequency range are allocated to the same storage space.

[0095] The adjustment module 54 is used to receive access operations to data blocks allocated in different storage spaces at preset time intervals, and adjust the storage location of the data blocks allocated in different storage spaces in different storage spaces according to the first access frequency corresponding to the access operation.

[0096] The aforementioned device allocates a group of data blocks to different storage spaces pre-defined in the cache according to a preset strategy. These different storage spaces include a first storage space that allows execution of a preset logging mechanism and a second storage space that allows synchronous updates with permanent storage. The preset strategy instructs data blocks with the same access frequency to be allocated to the same storage space. Access operations on the data blocks allocated to the different storage spaces are received at preset time intervals. The storage location of the data blocks allocated to the different storage spaces is adjusted based on the first access frequency corresponding to the access operations. Therefore, this addresses the technical problem in related technologies where the write-before-log mechanism used in storage systems records all operations on data in the cache in chronological order, failing to differentiate access frequencies for different types of data, leading to performance bottlenecks and resource waste. This achieves the technical effect of overcoming performance bottlenecks and saving storage system resources.

[0097] In an exemplary embodiment, the allocation module is further configured to divide the set of data blocks into multiple data block groups with different priorities according to the preset strategy; and allocate the multiple data block groups to the different storage spaces according to the priorities corresponding to the multiple data block groups respectively.

[0098] In an exemplary embodiment, the allocation module is further configured to obtain metadata information of each data block in the set of data blocks; and to divide the plurality of data block groups with different priorities based on the metadata information according to the preset strategy, wherein each data block group in the plurality of data block groups includes: data blocks in the set of data blocks that are in the same access frequency range.

[0099] In an exemplary embodiment, the allocation module is further configured to: determine the correspondence between the priority and the different storage spaces; determine a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space through the correspondence, wherein the plurality of data block groups includes: the first data block group and the second data block group; determine a first number of data blocks that can be recorded in the first storage space and a second number of data blocks that can be recorded in the second storage space; record the data blocks in the first data block group into the first storage space according to the first number, and record the data blocks in the second data block group into the second storage space according to the second number, so as to allocate the plurality of data block groups to the different storage spaces.

[0100] In an exemplary embodiment, the adjustment module is further configured to obtain the second access frequency corresponding to each of the w data blocks, wherein the w data blocks are data blocks allocated in each of the different storage spaces, and w is a positive integer; sort the w data blocks according to the second access frequency; and adjust the storage position of the data blocks allocated in the different storage spaces according to the sorted w data blocks.

[0101] In one exemplary embodiment, the adjustment module is further configured to obtain the target time corresponding to each of the w data blocks, wherein the target time is used to indicate the time interval between the last access time of each of the w data blocks and the current time; and to sort the w data blocks by the target time and the second access frequency.

[0102] In an exemplary embodiment, the adjustment module is further configured to normalize the target time and the second access frequency respectively; perform a weighted summation on the normalized target time and the normalized second access frequency to obtain a popularity value corresponding to each of the w data blocks; and sort the w data blocks according to the popularity value.

[0103] In an exemplary embodiment, the adjustment module is further configured to determine the adjustment order corresponding to the different storage spaces according to priority, so as to determine the third storage space and / or the fourth storage space in the different storage spaces, wherein the priority is the priority corresponding to the multiple data block groups allocated to the different storage spaces respectively, the group of data blocks includes the multiple data block groups, the third storage space is the storage space whose adjustment order is before each storage space, and the fourth storage space is the storage space whose adjustment order is after each storage space; and adjust the target data block group in the sorted w data blocks to the third storage space and / or the fourth storage space to adjust the storage position of the data blocks allocated in the different storage spaces.

[0104] In one exemplary embodiment, the adjustment module is further configured to exchange a third data block group in each storage space with a fourth data block group in the third storage space; and / or exchange a fifth data block group in each storage space with a sixth data block group in the fourth storage space; wherein the target data block group includes: the third data block group and / or the fifth data block group; the third data block group includes: a first preset proportion of data blocks ranked first among the sorted w data blocks in each storage space; the fifth data block group includes: a second preset proportion of data blocks ranked last among the sorted w data blocks in each storage space; the fourth data block group includes: a first preset proportion of data blocks ranked last among the sorted w data blocks in the third storage space; and the sixth data block group includes: a second preset proportion of data blocks ranked first among the sorted w data blocks in the fourth storage space.

[0105] In one exemplary embodiment, the apparatus further includes a log flushing and creation module, configured to: after adjusting the storage location of the data blocks allocated in the different storage spaces according to the first access frequency corresponding to the access operation; flush the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage; and create a second log file corresponding to the preset log mechanism, wherein the second log file is used to pre-write the read and write operations corresponding to the data blocks stored in the fifth storage space in the next time interval, the fifth storage space being the first storage space after the storage location of the stored data blocks has been adjusted.

[0106] In an exemplary embodiment, the apparatus further includes a replay module, configured to: before flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage, in the event of an abnormal event, restore the data blocks in the first storage space that have not been written to the permanent storage to the last consistent state through the replay mechanism corresponding to the first log file.

[0107] For a description of the features in the embodiment corresponding to the storage location adjustment device, please refer to the relevant description in the embodiment corresponding to the storage location adjustment method, which will not be repeated here.

[0108] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the storage location adjustment method.

[0109] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described storage location adjustment method embodiments when it is run.

[0110] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0111] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described storage location adjustment method embodiments.

[0112] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described storage location adjustment method embodiments.

[0113] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The above provides a detailed description of a storage location adjustment method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method of adjusting a storage location, characterized by, The method comprises: allocating a group of data blocks to different storage spaces in a cache according to a preset policy, wherein the different storage spaces comprise a first storage space allowing a preset log mechanism to be performed and a second storage space allowing synchronization update with a permanent storage, and the preset policy is used to indicate that data blocks in a same access frequency range are allocated to a same storage space; receiving an access operation on the data blocks allocated to the different storage spaces according to a preset time interval, and adjusting storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to a first access frequency corresponding to the access operation, wherein allocating a group of data blocks to different storage spaces in a cache according to a preset policy comprises: dividing the group of data blocks into a plurality of data block groups with different priorities according to the preset policy; and allocating the plurality of data block groups to the different storage spaces according to priorities corresponding to the plurality of data block groups, wherein dividing the group of data blocks into a plurality of data block groups with different priorities according to the preset policy comprises: obtaining metadata information of each data block in the group of data blocks; and dividing the plurality of data block groups with different priorities based on the metadata information according to the preset policy, wherein each data block group in the plurality of data block groups comprises data blocks in the group of data blocks in a same access frequency range, wherein the access frequency of each data block is determined by an initial access count in the metadata information, and each data block in the group of data blocks is divided into the plurality of data block groups according to a size relationship between the access frequency and a plurality of frequency thresholds or a relationship between a plurality of access frequency ranges, wherein allocating the plurality of data block groups to the different storage spaces according to priorities corresponding to the plurality of data block groups comprises: determining a correspondence relationship between the priorities and the different storage spaces; determining a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space through the correspondence relationship, wherein the plurality of data block groups comprise the first data block group and the second data block group; determining a first number of data blocks allowed to be recorded in the first storage space and a second number of data blocks allowed to be recorded in the second storage space; allocating data blocks in the first data block group to the first storage space according to the first number and allocating data blocks in the second data block group to the second storage space according to the second number to allocate the plurality of data block groups to the different storage spaces.

2. The storage location adjustment method according to claim 1, wherein adjusting the storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation comprises: obtaining second access frequencies corresponding to w data blocks, wherein the w data blocks are data blocks allocated to each storage space in the different storage spaces, and w is a positive integer; sorting the w data blocks according to the second access frequencies; and adjusting the storage positions of the w data blocks in the different storage spaces according to the second access frequencies. Adjust storage positions of the data blocks allocated in the different storage spaces in the different storage spaces through the sorted w data blocks.

3. The storage location adjustment method according to claim 2, wherein The sorting of the w data blocks through the second access frequencies comprises: obtaining target times corresponding to the w data blocks respectively, wherein the target time is used to indicate a time interval between a time when each data block in the w data blocks is last accessed and a current time; sorting the w data blocks through the target times and the second access frequencies.

4. The storage location adjustment method according to claim 3, wherein The sorting of the w data blocks through the target times and the second access frequencies comprises: normalizing the target times and the second access frequencies respectively; performing weighted summation on the normalized target times and the normalized second access frequencies to obtain a hotness value corresponding to each data block in the w data blocks; sorting the w data blocks through the hotness values.

5. The storage location adjustment method according to claim 2, wherein Adjusting storage positions of the data blocks allocated in the different storage spaces in the different storage spaces through the sorted w data blocks comprises: determining an adjustment order corresponding to the different storage spaces according to priorities to determine a third storage space and / or a fourth storage space in the different storage spaces, wherein the priority is a priority corresponding to a plurality of data block groups allocated in the different storage spaces respectively, the data block group comprises the plurality of data block groups, the third storage space is a storage space whose adjustment order is in front of each storage space, and the fourth storage space is a storage space whose adjustment order is behind each storage space; adjusting a target data block group in the sorted w data blocks into the third storage space and / or the fourth storage space to adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces.

6. The storage location adjustment method according to claim 5, wherein Adjusting a target data block group in the sorted w data blocks into the third storage space and / or the fourth storage space to adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces comprises: exchanging a third data block group in each storage space with a fourth data block group in the third storage space, and / or exchanging a fifth data block group in each storage space with a sixth data block group in the fourth storage space; wherein the target data block group comprises the third data block group and / or the fifth data block group, the third data block group comprises a first preset proportion of data blocks in front in the sorted w data blocks in each storage space, and the fifth data block group comprises a second preset proportion of data blocks behind in the sorted w data blocks in each storage space; the fourth data block group comprises a first preset proportion of data blocks behind in the sorted w data blocks in the third storage space, and the sixth data block group comprises a second preset proportion of data blocks in front in the sorted w data blocks in the fourth storage space.

7. The storage location adjustment method according to claim 1, wherein After adjusting the storage location of the data block allocated in the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation, the method further comprises: flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage; creating a second log file corresponding to the preset log mechanism, wherein the second log file is used to pre-write the corresponding read-write operation of the data block stored in the fifth storage space in the next time interval, and the fifth storage space is the first storage space after the storage location of the stored data block is adjusted.

8. The storage location adjustment method according to claim 7, wherein Before flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage, the method further comprises: In the case of an abnormal event, restoring the data block in the first storage space that is not written to the permanent storage to the last consistent state through the replay mechanism corresponding to the first log file.

9. An adjustment device for a storage location, characterized by Comprise: The allocation module is configured to allocate a group of data blocks to different storage spaces pre-divided in the cache according to a preset strategy, wherein the different storage spaces include a first storage space allowing a preset log mechanism to be executed and a second storage space allowing synchronization update with a permanent storage, and the preset strategy is used to indicate that data blocks in the same access frequency range are divided into the same storage space. The adjustment module is configured to receive access operations on the data blocks allocated in the different storage spaces according to a preset time interval, and adjust the storage location of the data blocks allocated in the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation, Wherein, the allocation module is further configured to divide the group of data blocks into a plurality of data block groups with different priorities according to the preset strategy; and allocate the plurality of data block groups to the different storage spaces according to the priorities corresponding to the plurality of data block groups, respectively, wherein The allocation module is further configured to obtain metadata information of each data block in the group of data blocks; and divide the plurality of data block groups with different priorities based on the metadata information according to the preset strategy, wherein each data block group in the plurality of data block groups includes data blocks in the same access frequency range in the group of data blocks, wherein the access frequency of each data block is determined by the initial access count in the metadata information, and each data block in the group of data blocks is divided into a plurality of data block groups according to the size relationship between the access frequency and the preset plurality of frequency thresholds, or the relationship between the preset plurality of access frequency ranges, The allocation module is further configured to determine the correspondence between the priority and the different storage spaces; determine a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space according to the correspondence, wherein the plurality of data block groups include the first data block group and the second data block group; determine the first number of data blocks allowed to be recorded in the first storage space and the second number of data blocks allowed to be recorded in the second storage space; The data blocks in the first data block group are recorded into the first storage space according to the first number, and the data blocks in the second data block group are recorded into the second storage space according to the second number, so as to allocate the multiple data block groups into the different storage spaces.

10. An electronic device, comprising: Comprise: a memory for storing a computer program; a processor for implementing the steps of the storage location adjustment method as claimed in any one of claims 1 to 8 when executing the computer program.

11. A computer readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the storage location adjustment method as claimed in any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the storage location adjustment method as claimed in any one of claims 1 to 8.

Images