Storage position adjusting method and device, equipment, storage medium and program product

Abstract

The invention discloses a storage position adjusting method and device, equipment, a storage medium and a program product, and relates to the technical field of storage, and the method comprises the steps: distributing a group of data blocks into different storage spaces divided in a cache in advance according to a preset strategy, the different storage spaces comprise a first storage space allowing execution of a preset log mechanism and a second storage space allowing synchronous updating with permanent storage; and receiving access operations on the distributed data blocks in the different storage spaces according to a preset time interval, and adjusting the storage positions of the distributed data blocks in the different storage spaces in the different storage spaces through first access frequencies corresponding to the access operations. Therefore, the problems of performance bottleneck and resource waste caused by the fact that a pre-write log mechanism used by the storage system records all operations on the data in the cache in a time sequence and does not distinguish the access frequency of different types of data can be solved, and the effects of breaking the performance bottleneck and saving the resources of the storage system are achieved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a method, apparatus, device, storage medium, and program product for adjusting a storage position. Background Art

[0002] Storage systems in related technologies, such as database management systems, distributed file systems, and critical business platforms, commonly utilize a write-ahead logging (WAL) mechanism to ensure data consistency and integrity in the event of a system failure and support rapid recovery. The basic principle of this mechanism is to record changes in a log file before officially writing the data, allowing data recovery by replaying the log file in the event of an anomaly. However, mainstream WAL mechanisms typically record all operations in chronological order, without distinguishing between the access frequencies of different types of data. This leads to significant performance bottlenecks and resource waste, especially when processing frequently accessed hot data.

[0003] Regarding related technologies, the write-ahead log mechanism used by the storage system records all operations on the data in the cache in chronological order, without distinguishing the access frequency of different types of data, resulting in performance bottlenecks and resource waste, which have not yet been effectively resolved. Summary of the Invention

[0004] The present application provides a storage location adjustment method, apparatus, device, storage medium and program product to at least solve the problem that the write-ahead log mechanism used by the storage system in the related art records all operations on the data in the cache in chronological order, does not distinguish the access frequency of different types of data, and leads to performance bottlenecks and resource waste.

[0005] The present application provides a storage position adjustment method, comprising: allocating a group of data blocks to different storage spaces pre-divided in a cache according to a preset strategy, wherein the different storage spaces include: a first storage space allowing execution of a preset log mechanism and a second storage space allowing synchronization updates with permanent storage, the preset strategy being used to indicate that data blocks within the same access frequency range are divided into the same storage space; receiving access operations to the data blocks allocated to the different storage spaces at preset time intervals, and adjusting the storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to the first access frequencies corresponding to the access operations.

[0006] The present application also provides a storage position adjustment device, comprising: an allocation module, for allocating a group of data blocks to different storage spaces pre-divided in a cache according to a preset strategy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset strategy is used to indicate that data blocks in the same access frequency range are divided into the same storage space; an adjustment module, for receiving access operations to the data blocks allocated to the different storage spaces at preset time intervals, and adjusting the storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned storage position adjustment methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned storage position adjustment methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned storage location adjustment methods when the computer program is executed by a processor.

[0010] Through this application, a group of data blocks are allocated to different storage spaces pre-divided in the cache according to a preset strategy, wherein the different storage spaces include: a first storage space that allows the execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset strategy is used to indicate that data blocks in the same access frequency range are divided into the same storage space; access operations to the data blocks allocated in the different storage spaces are received according to a preset time interval, and the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces are adjusted according to the first access frequency corresponding to the access operation. Therefore, the problem that the write-before log mechanism used by the storage system in the related art records all operations on the data in the cache in chronological order and does not distinguish between the access frequencies of different types of data, resulting in performance bottlenecks and resource waste can be solved, thereby achieving the technical effect of breaking through performance bottlenecks and saving storage system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a hardware structure block diagram of a computer terminal for a storage position adjustment method according to an embodiment of the present application;

[0013] Figure 2 is a flow chart of a method for adjusting a storage position according to an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a storage space according to a method for adjusting a storage position according to an embodiment of the present application;

[0015] Figure 4 is another flow chart of a method for adjusting a storage position according to an embodiment of the present application;

[0016] Figure 5 This is a structural block diagram of a storage position adjustment device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the storage location adjustment method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0021] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure diagram of a computer terminal for a method of adjusting a storage location according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0022] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the storage position adjustment method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0023] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0024] Figure 2 FIG. 1 is a flow chart of a method for adjusting a storage location according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0025] Step S202: Allocating a group of data blocks to different pre-divided storage spaces in the cache according to a preset policy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset policy is used to indicate that data blocks with the same access frequency range are allocated to the same storage space;

[0026] Step S204: receiving access operations to the data blocks allocated to the different storage spaces at preset time intervals, and adjusting storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to first access frequencies corresponding to the access operations.

[0027] Through the above steps, a group of data blocks are allocated to different storage spaces pre-divided in the cache according to a preset strategy, wherein the different storage spaces include: a first storage space that allows the execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset strategy is used to indicate that data blocks in the same access frequency range are divided into the same storage space; access operations to the data blocks allocated in the different storage spaces are received according to a preset time interval, and the storage positions of the data blocks allocated in the different storage spaces are adjusted according to the first access frequency corresponding to the access operation. Therefore, the problem that the write-before log mechanism used by the storage system in the related art records all operations on the data in the cache in chronological order and does not distinguish between the access frequencies of different types of data, resulting in performance bottlenecks and resource waste can be solved, thereby achieving the technical effect of breaking through performance bottlenecks and saving storage system resources.

[0028] An embodiment of the present application provides a method for adjusting a storage position, and the method is described in detail in conjunction with the execution flow of the method for adjusting a storage position.

[0029] In an exemplary embodiment, a group of data blocks are allocated to different storage spaces pre-divided in a cache according to a preset strategy, including: dividing the group of data blocks into multiple data block groups with different priorities according to the preset strategy; and allocating the multiple data block groups to the different storage spaces according to the priorities corresponding to the multiple data block groups.

[0030] This application can optimize data storage, improve system performance, and enhance stability and fault tolerance, which has important practical value for large-scale distributed storage and database systems.

[0031] Optionally, dividing the group of data blocks into multiple data block groups with different priorities according to the preset strategy includes: obtaining metadata information of each data block in the group of data blocks; dividing the multiple data block groups with different priorities based on the metadata information according to the preset strategy, wherein each data block group in the multiple data block groups includes: data blocks in the group of data blocks that are in the same access frequency range.

[0032] The metadata information includes: identification information, initial access count, access time, category, etc. of each data block in a group of data blocks. Furthermore, the access frequency of each data block can be determined based on the initial access count in the metadata information. Each data block in a group of data blocks can be divided into multiple data block groups based on the relationship between the access frequency and multiple preset frequency thresholds, or the relationship between multiple preset access frequency ranges. For example, the multiple data block groups include: a high-frequency cache data block group, into which data blocks with access frequencies greater than frequency threshold A1 can be divided; a medium-frequency cache data block group, into which data blocks with access frequencies between frequency threshold A1 and frequency threshold B1 can be divided; and so on.

[0033] Optionally, allocating the multiple data block groups to the different storage spaces according to the priorities corresponding to the multiple data block groups respectively includes: determining the correspondence between the priorities and the different storage spaces; determining a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space through the correspondence, wherein the multiple data block groups include: the first data block group and the second data block group; determining a first number of data blocks allowed to be recorded in the first storage space and a second number of data blocks allowed to be recorded in the second storage space; recording the data blocks in the first data block group in the first storage space according to the first number, and recording the data blocks in the second data block group in the second storage space according to the second number, so as to allocate the multiple data block groups to the different storage spaces.

[0034] For example, the first data block group can be a high-frequency cache data block group, which is stored in a first storage space that can execute a preset log mechanism (such as a write-ahead log mechanism), and all read and write operations on the cached data blocks in the first storage space are pre-written into the log corresponding to the preset log mechanism; the second data block group can be a medium-frequency cache data block group, which is stored in a second storage space that is updated synchronously with the permanent storage, and all read and write operations on the cached data blocks in the second storage space are updated synchronously with the permanent storage.

[0035] Through this application, the strategy of dynamically allocating data blocks to different storage spaces according to priority can not only significantly improve system performance, but also optimize resource utilization and enhance the security and reliability of data management. It has important technical significance and practical value for distributed database systems, data-intensive applications and data storage management in high-load environments.

[0036] In an exemplary embodiment, adjusting the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces by the first access frequency corresponding to the access operation includes: obtaining second access frequencies corresponding to w data blocks respectively, wherein the w data blocks are the data blocks allocated in each storage space in the different storage spaces, and w is a positive integer; sorting the w data blocks by the second access frequency; and adjusting the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces by the sorted w data blocks.

[0037] The second access frequency is the access frequency of each data block in the w data blocks within a preset time interval.

[0038] It should be noted that the access frequency in this application refers to the access frequency.

[0039] Furthermore, the w data blocks are sorted according to the second access frequency, including: obtaining target times corresponding to the w data blocks respectively, wherein the target time is used to indicate the time interval between the last access time of each of the w data blocks and the current time; and sorting the w data blocks according to the target time and the second access frequency.

[0040] Specifically, the w data blocks are sorted by the target time and the second access frequency, including: normalizing the target time and the second access frequency respectively; taking a weighted sum of the normalized target time and the normalized second access frequency to obtain a heat value corresponding to each data block in the w data blocks; and sorting the w data blocks by the heat value.

[0041] In other words, the heat value can be calculated by the following formula:

[0042] .

[0043] in, Indicates the heat value corresponding to each data block in w data blocks, Represents the normalization function, which can be selected as min-max normalization. represents the second access frequency of each data block in the w data blocks, Indicates the target time, and Represents weight.

[0044] Through this application, by normalizing and weighted summing the target time and access frequency to generate a heat value, and then sorting the data blocks, the intelligence level of cache management can be significantly improved, data access performance can be optimized, and the needs of different business scenarios can be adapted. This reflects the importance of dynamic resource management and data heat perception, and has important technical significance for building a high-performance and highly adaptable distributed storage system.

[0045] In an exemplary embodiment, adjusting the storage positions of the data blocks allocated in the different storage spaces by using the sorted w data blocks includes: determining the adjustment order corresponding to the different storage spaces according to the priority to determine the third storage space and / or the fourth storage space in the different storage spaces, wherein the priority is the priority corresponding to the multiple data block groups allocated to the different storage spaces, the group of data blocks includes the multiple data block groups, the third storage space is the storage space whose adjustment order is before each storage space, and the fourth storage space is the storage space whose adjustment order is after each storage space; adjusting the target data block group in the sorted w data blocks to the third storage space and / or the fourth storage space to adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces.

[0046] For example, if the priority of the high-frequency cache data block group is higher than that of the medium-frequency data block group, the storage space for storing the high-frequency cache data block group is adjusted before the storage space for storing the medium-frequency data block group.

[0047] Furthermore, the target data block group among the sorted w data blocks is adjusted to the third storage space and / or the fourth storage space to adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces, including: exchanging the third data block group in each storage space with the fourth data block group in the third storage space; and / or exchanging the fifth data block group in each storage space with the sixth data block group in the fourth storage space; wherein the target data block group includes: the third data block group and / or the fifth data block group; the third data block group includes: the data blocks with a first preset proportion sorted in front of the w data blocks sorted in each storage space, the fifth data block group includes: the data blocks with a second preset proportion sorted in back of the w data blocks sorted in each storage space; the fourth data block group includes: the data blocks with a first preset proportion sorted in back of the w data blocks sorted in the third storage space, and the sixth data block group includes: the data blocks with a second preset proportion sorted in front of the w data blocks sorted in the fourth storage space.

[0048] For example, there are high-frequency storage space, medium-frequency storage space, and low-frequency storage space. The last 30% of the data blocks in the w data blocks sorted in the high-frequency storage space are exchanged with the first 30% of the data blocks in the w data blocks sorted in the medium-frequency storage space, and the last 30% of the data blocks in the w data blocks sorted in the medium-frequency storage space are exchanged with the first 30% of the data blocks in the w data blocks sorted in the low-frequency storage space. The last 30% of the data blocks in the w data blocks sorted in the low-frequency storage space can be exchanged with data blocks that are not stored in different pre-divided storage spaces. For example, a group of data blocks that are divided into a high-frequency cache data block group may not be fully stored in the high-frequency storage space, but are stored in a data disk. In this case, the last 30% of the data blocks in the w data blocks sorted in the low-frequency storage space can be exchanged with data in the data disk.

[0049] For another example, only the heat values ​​of the w data blocks in the first storage space may be calculated and then sorted, while the other storage spaces including the second storage space are sorted based solely on the second access frequency of each of the w data blocks. The positions of the data blocks in the different storage spaces are then swapped according to the storage space adjustment order, the first preset ratio, and the second preset ratio.

[0050] In an exemplary embodiment, after adjusting the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces through the first access frequency corresponding to the access operation, the method further includes: flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage; creating a new second log file corresponding to the preset log mechanism, wherein the second log file is used to pre-write the read and write operations corresponding to the data blocks stored in the fifth storage space within the next time interval, and the fifth storage space is the first storage space after the storage position of the stored data blocks is adjusted.

[0051] Furthermore, before flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage, the method also includes: in the event of an abnormal event, restoring the data blocks in the first storage space that have not been written to the permanent storage to the last consistent state through the replay mechanism corresponding to the first log file.

[0052] Through this application, the WAL mechanism's regular data flushing, log file reconstruction, and data recovery strategy under abnormal events can effectively ensure data persistence and consistency, while optimizing system performance and resource utilization, improving fault recovery efficiency, simplifying operation and maintenance work, and enhancing the system's adaptability and stability. It has a significant technical effect on the high efficiency and high reliability of distributed storage systems.

[0053] In order to better understand the process of the above-mentioned storage position adjustment method, the implementation process of the above-mentioned storage position adjustment method is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of this application.

[0054] The optional embodiment of the present application is mainly applied to the data caching problem of the storage system. By demarcating hot data and selectively caching data, combined with the write-ahead log mechanism, the performance of the storage system can be significantly improved.

[0055] Storage systems in related technologies, such as database management systems, distributed file systems, and critical business platforms, generally employ a write-ahead logging mechanism to ensure data consistency and integrity in the event of a system failure and support rapid recovery. The basic principle of this mechanism is to record changes in a log file before officially writing the data, allowing data recovery by replaying the log file in the event of an anomaly. However, the current mainstream WAL mechanism typically records all operations in chronological order, without distinguishing between the access frequencies of different types of data. This leads to significant performance bottlenecks and resource waste, especially when processing frequently accessed hot data.

[0056] Specifically, hot data blocks, due to frequent access and modification, generate a large number of repeated log entries, leading to severe write amplification and a rapid increase in storage resource usage. This redundant write not only wastes disk input / output (I / O), but also increases system burden and reduces overall throughput. Secondly, traditional caching mechanisms are typically not optimized to dynamically evolve with access patterns, failing to accurately identify and prioritize hot data. This results in low cache hit rates, reduced read performance, and indirectly increased burden on the logging system. Furthermore, during system failure recovery, the WAL mechanism often requires the sequential replay of all log records to restore to a consistent state, even if the majority of data has already been persisted and remains unchanged. This "one-size-fits-all" replay strategy is time-consuming and inefficient, failing to meet modern requirements for high availability and fast recovery.

[0057] Therefore, the existing WAL mechanism has room for improvement in handling hot data and improving fault recovery efficiency, and there is an urgent need to introduce a more adaptive log and cache collaborative optimization solution.

[0058] In order to solve the above-mentioned defects, an optional embodiment of the present application proposes a data caching method based on hot spot data and pre-write logs in a distributed storage system (equivalent to the storage position adjustment method in the above-mentioned embodiment), including: setting the initial configuration information of the data block, and dividing the data block into 2-4 priorities. Then, the upper-layer business is run to collect the number of times each data block is used. According to the frequency of data use, different iteration strategies are adopted to adjust the data cache. The cached data is read and written by means of a pre-write log. After the next iteration, the data in the pre-write log is refreshed according to the re-determined hot spot data. If data loss occurs along the way, data recovery is performed according to the pre-write log to ensure data security and read and write efficiency.

[0059] Specifically, the basic ideas of the optional embodiments of this application are as follows:

[0060] 1) Initialization phase: Set the initial configuration information and priority division of the data blocks.

[0061] When the system boots or when a new data block is written, basic configuration information is set for each data block in a group, such as the data block identification (ID), initial access count, and access timestamp. Subsequently, all data blocks are divided into two to four different priority levels based on default rules (such as preset access frequency, data type, or application tags). For example, these can be divided into categories such as "high-frequency access," "medium-frequency access," "low-frequency access," and "initially unknown" to facilitate subsequent dynamic adjustment.

[0062] 2) Run the business and dynamically collect the access frequency of data blocks.

[0063] The system begins normal operation, executing upper-level services (such as database operations or file reads and writes). During this process, access information for each data block is collected in real time or periodically, including read counts, write frequency, and write time. This data is used to assess the current data block's "hotness" and provide a basis for iteratively adjusting strategies.

[0064] 3) Iterate based on access frequency.

[0065] Based on the access frequency of the collected data blocks, the priority of each data block is dynamically adjusted using a specified iteration strategy. Frequently accessed data blocks are prioritized and assigned a higher cache weight; while less frequently accessed data blocks may be downgraded or even removed from the cache, thereby improving the overall cache hit rate.

[0066] 4) Combine the write-ahead log mechanism to manage the reading and writing of cached data.

[0067] The system uses a write-ahead log (WAL) mechanism to manage selected cached data blocks. All read and write operations on cached data are first written to the WAL file, ensuring traceability and recoverability. This not only enhances system fault tolerance but also avoids the frequent I / O overhead associated with direct disk writes.

[0068] 5) Iterative update and cache flushing strategy.

[0069] At the start of the next iteration, the system re-identifies hot data based on new data access patterns. For data blocks that are no longer hot, changes in their WALs are flushed to permanent storage (e.g., disk or object storage) and removed from the cache, maintaining efficient cache space utilization. At the same time, hot data at the current stage is prioritized for retention in the cache.

[0070] 6) Fault recovery mechanism, using write-ahead logs to ensure data security.

[0071] If a system crash, power outage, or other anomaly occurs during this process, the system can restore data not yet written to persistent storage to its last consistent state through the WAL file replay mechanism. This mechanism ensures that even if cached data has not yet been flushed, recovery can be completed through logs, significantly improving the system's fault tolerance and data security.

[0072] Furthermore, in combination with the above basic ideas, the implementation scheme of the optional embodiment of the present application is as follows: Figure 3 Shown, including:

[0073] Step S31: Initialize the data space and set the initial configuration information and priority division of the data blocks.

[0074] In the distributed storage system, metadata information of the data block is set, and the ID, initial access count, access time and category of the data block are recorded respectively. The data blocks are divided into four types: high-frequency access blocks, medium-frequency access blocks, low-frequency word access blocks, and initial data blocks. The cache space of high-frequency cache data is set to R1, and L1 data blocks are recorded. The cache space of medium-frequency cache data is set to R2, and L2 data blocks are recorded. Figure 4 shown.

[0075] Step S32: Run the service and dynamically collect the access frequency and time of data blocks.

[0076] Run the upper-layer business for a period of time, record the frequency of data access, and place the first L1+L2 data blocks in the R1 and R2 spaces respectively based on the access frequency.

[0077] Step S33: performing data location exchange based on access frequency.

[0078] Set a fixed time interval T (equivalent to the preset time interval in the above embodiment), and after time T, obtain the access frequency of the R1 space data block and the time interval between the last access time point and the current time point , and then sort the data blocks by normalization. The heat value based on which the sorting is based is calculated as follows:

[0079] .

[0080] in, =0.7, =0.3( and Parameters can be adjusted according to actual conditions). Represents the normalization function, here refers to min-max normalization:

[0081] .

[0082] The last 30% of the sorted data in R1 is swapped with the first 30% of the data in R2, which is sorted by access frequency. The last 30% of the data in R2 is swapped with the frequently accessed data on the data disk.

[0083] Step S34: managing the reading and writing of cached data in conjunction with the write-before log mechanism.

[0084] During the T interval, data in the R1 cache is first written to the WAL file. Data written to the R2 space is updated simultaneously in memory and on disk. This ensures that operations are traceable and recoverable, while also ensuring that the amount of data being operated on is not excessive.

[0085] Step S35: Iterative update and cache flushing strategy.

[0086] At the beginning of the next iteration cycle, that is, after the position exchange in step S33 is completed, the data in the WAL log is flushed to the disk, and a new WAL is created to record the new round of data operations.

[0087] Step S36: Execute the fault recovery mechanism and use the write-ahead log to ensure data security.

[0088] When the system crashes due to incomplete data flushing, the next time data storage is restarted, the WAL data is first flushed to update the data to the latest version, and then the iteration is restarted starting from step S32.

[0089] Optionally, the data space can be refined to correspond to multiple WALs to improve reading and writing efficiency; normalization The function can be another normalization function, such as Z-Score normalization.

[0090] In summary, the optional embodiment of the present application first sets the initial configuration information of the data block and divides the data block into 2-4 priorities. Then, the upper-layer business is run to collect the number of times each data block is used. According to the frequency of use of the data, an iterative strategy is adopted to adjust the data cache. The cached data is read and written by means of a write-before log. After the next iteration, the data in the write-before log is flushed based on the re-determined hot data. If data loss occurs along the way, data recovery is performed based on the write-before log to ensure data security and read and write efficiency.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0092] This embodiment also provides a storage position adjustment device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0093] Figure 5 is a structural block diagram of a storage position adjustment device according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0094] an allocation module 52 configured to allocate a group of data blocks to different pre-divided storage spaces in the cache according to a preset policy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronization updates with the permanent storage; and wherein the preset policy is configured to instruct data blocks within the same access frequency range to be allocated to the same storage space.

[0095] The adjustment module 54 is used to receive access operations to the data blocks allocated in the different storage spaces at preset time intervals, and adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces according to the first access frequencies corresponding to the access operations.

[0096] Through the above-mentioned device, a group of data blocks are allocated to different storage spaces pre-divided in the cache according to a preset strategy, wherein the different storage spaces include: a first storage space that allows the execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset strategy is used to indicate that data blocks in the same access frequency range are divided into the same storage space; access operations to the data blocks allocated in the different storage spaces are received according to a preset time interval, and the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces are adjusted according to the first access frequency corresponding to the access operation. Therefore, the technical problem that the write-before log mechanism used by the storage system in the related art records all operations on the data in the cache in chronological order without distinguishing the access frequencies of different types of data, resulting in performance bottlenecks and resource waste, can be solved, thereby achieving the technical effect of breaking through performance bottlenecks and saving storage system resources.

[0097] In an exemplary embodiment, the allocation module is further configured to divide the group of data blocks into multiple data block groups with different priorities according to the preset strategy; and allocate the multiple data block groups to the different storage spaces according to the priorities corresponding to the multiple data block groups.

[0098] In an exemplary embodiment, the allocation module is further used to obtain metadata information of each data block in the group of data blocks; and divide the multiple data block groups with different priorities based on the metadata information according to the preset strategy, wherein each data block group in the multiple data block groups includes: data blocks in the group of data blocks that are in the same access frequency range.

[0099] In an exemplary embodiment, the allocation module is further used to determine the correspondence between the priorities and the different storage spaces; determine a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space through the correspondence, wherein the multiple data block groups include: the first data block group and the second data block group; determine a first number of data blocks allowed to be recorded in the first storage space and a second number of data blocks allowed to be recorded in the second storage space; record the data blocks in the first data block group into the first storage space according to the first number, and record the data blocks in the second data block group into the second storage space according to the second number, so as to allocate the multiple data block groups to the different storage spaces.

[0100] In an exemplary embodiment, the adjustment module is further used to obtain second access frequencies corresponding to w data blocks, respectively, wherein the w data blocks are data blocks allocated to each storage space in the different storage spaces, and w is a positive integer; sort the w data blocks according to the second access frequencies; and adjust the storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to the sorted w data blocks.

[0101] In an exemplary embodiment, the adjustment module is further used to obtain target times corresponding to the w data blocks respectively, wherein the target time is used to indicate the time interval between the last access time of each of the w data blocks and the current time; and the w data blocks are sorted according to the target time and the second access frequency.

[0102] In an exemplary embodiment, the adjustment module is further used to normalize the target time and the second access frequency respectively; perform weighted summation on the normalized target time and the normalized second access frequency to obtain a heat value corresponding to each of the w data blocks; and sort the w data blocks according to the heat value.

[0103] In an exemplary embodiment, the adjustment module is also used to determine the adjustment order corresponding to the different storage spaces according to the priority to determine the third storage space and / or the fourth storage space in the different storage spaces, wherein the priority is the priority corresponding to the multiple data block groups allocated to the different storage spaces, the group of data blocks includes the multiple data block groups, the third storage space is the storage space whose adjustment order is before each storage space, and the fourth storage space is the storage space whose adjustment order is after each storage space; the target data block group in the sorted w data blocks is adjusted to the third storage space and / or the fourth storage space to adjust the storage position of the data blocks allocated in the different storage spaces in the different storage spaces.

[0104] In an exemplary embodiment, the adjustment module is further used to exchange the third data block group in each storage space with the fourth data block group in the third storage space; and / or exchange the fifth data block group in each storage space with the sixth data block group in the fourth storage space; wherein the target data block group includes: the third data block group and / or the fifth data block group; the third data block group includes: the first preset proportion of data blocks sorted in front of the w data blocks sorted in each storage space, and the fifth data block group includes: the second preset proportion of data blocks sorted in the back of the w data blocks sorted in each storage space; the fourth data block group includes: the first preset proportion of data blocks sorted in the back of the w data blocks sorted in the third storage space, and the sixth data block group includes: the second preset proportion of data blocks sorted in front of the w data blocks sorted in the fourth storage space.

[0105] In an exemplary embodiment, the device also includes a log flushing and new creation module, which is used to: flush the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage after adjusting the storage position of the data blocks allocated in the different storage spaces in the different storage spaces according to the first access frequency corresponding to the access operation; and create a second log file corresponding to the preset log mechanism, wherein the second log file is used to pre-write the read and write operations corresponding to the data blocks stored in the fifth storage space in the next time interval, and the fifth storage space is the first storage space after the storage position of the stored data blocks is adjusted.

[0106] In an exemplary embodiment, the device also includes a replay module for flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage: in the event of an abnormal event, the data blocks in the first storage space that have not been written to the permanent storage are restored to the last consistent state through the replay mechanism corresponding to the first log file.

[0107] For the description of the features in the embodiment corresponding to the storage position adjustment device, reference can be made to the relevant description of the embodiment corresponding to the storage position adjustment method, which will not be repeated here.

[0108] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned storage position adjustment method embodiments.

[0109] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned storage location adjustment method embodiments when running.

[0110] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0111] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned storage location adjustment method embodiments are implemented.

[0112] An embodiment of the present application further provides another computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned storage location adjustment method embodiments are implemented.

[0113] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The above is a detailed introduction to a storage position adjustment method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for adjusting a storage position, characterized in that: include: Allocating a group of data blocks to different pre-divided storage spaces in the cache according to a preset policy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset policy is used to indicate that data blocks within the same access frequency range are allocated to the same storage space; Access operations to the data blocks allocated to the different storage spaces are received at preset time intervals, and storage positions of the data blocks allocated to the different storage spaces in the different storage spaces are adjusted according to a first access frequency corresponding to the access operations.

2. The method for adjusting the storage position according to claim 1, wherein: Allocate a set of data blocks to different pre-divided storage spaces in the cache according to a preset strategy, including: Dividing the group of data blocks into a plurality of data block groups with different priorities according to the preset strategy; The multiple data block groups are allocated to the different storage spaces according to their respective corresponding priorities.

3. The method for adjusting the storage position according to claim 2, wherein: Dividing the group of data blocks into a plurality of data block groups with different priorities according to the preset strategy includes: Obtaining metadata information of each data block in the set of data blocks; The plurality of data block groups with different priorities are divided based on the metadata information according to the preset strategy, wherein each of the plurality of data block groups includes: data blocks in the group of data blocks that are in the same access frequency range.

4. The method for adjusting the storage position according to claim 2, wherein: Allocating the plurality of data block groups to the different storage spaces according to the priorities respectively corresponding to the plurality of data block groups includes: Determining a correspondence between the priorities and the different storage spaces; Determining a first data block group corresponding to the first storage space and a second data block group corresponding to the second storage space according to the corresponding relationship, wherein the plurality of data block groups include: the first data block group and the second data block group; determining a first number of data blocks allowed to be recorded in the first storage space and a second number of data blocks allowed to be recorded in the second storage space; The data blocks in the first data block group are recorded into the first storage space according to the first quantity, and the data blocks in the second data block group are recorded into the second storage space according to the second quantity, so as to allocate the multiple data block groups into the different storage spaces.

5. The method for adjusting the storage position according to claim 1, wherein: Adjusting the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces according to the first access frequencies corresponding to the access operations includes: Obtaining second access frequencies corresponding to w data blocks, respectively, wherein the w data blocks are data blocks allocated to each storage space in the different storage spaces, and w is a positive integer; sorting the w data blocks according to the second access frequency; The storage positions of the data blocks allocated in the different storage spaces are adjusted using the sorted w data blocks.

6. The method for adjusting the storage position according to claim 5, characterized in that: Sorting the w data blocks according to the second access frequency includes: Obtain target times corresponding to the w data blocks respectively, wherein the target time is used to indicate the time interval between the last access time of each data block in the w data blocks and the current time; The w data blocks are sorted according to the target time and the second access frequency.

7. The method for adjusting the storage position according to claim 6, wherein: Sorting the w data blocks according to the target time and the second access frequency includes: Normalizing the target time and the second access frequency respectively; Performing a weighted summation on the normalized target time and the normalized second access frequency to obtain a heat value corresponding to each of the w data blocks; The w data blocks are sorted according to the heat values.

8. The method for adjusting the storage position according to claim 5, wherein: Adjusting storage positions of the data blocks allocated in the different storage spaces by using the sorted w data blocks includes: determining an adjustment order corresponding to the different storage spaces according to priorities, so as to determine a third storage space and / or a fourth storage space in the different storage spaces, wherein the priorities are priorities respectively corresponding to a plurality of data block groups allocated to the different storage spaces, the group of data blocks includes the plurality of data block groups, the third storage space is a storage space that precedes each of the storage spaces in an adjustment order, and the fourth storage space is a storage space that follows each of the storage spaces in an adjustment order; The target data block group in the sorted w data blocks is adjusted to the third storage space and / or the fourth storage space, so as to adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces.

9. The method for adjusting the storage position according to claim 8, wherein: Adjusting the target data block group in the sorted w data blocks to the third storage space and / or the fourth storage space to adjust the storage positions of the data blocks allocated in the different storage spaces in the different storage spaces includes: exchanging the third data block group in each storage space with the fourth data block group in the third storage space; and / or exchanging the fifth data block group in each storage space with the sixth data block group in the fourth storage space; The target data block group includes: the third data block group and / or the fifth data block group; the third data block group includes: a first preset proportion of data blocks that are sorted first among the w data blocks sorted in each storage space; the fifth data block group includes: a second preset proportion of data blocks that are sorted last among the w data blocks sorted in each storage space; The fourth data block group includes: a first preset proportion of data blocks sorted later among the w data blocks sorted in the third storage space; and the sixth data block group includes: a second preset proportion of data blocks sorted earlier among the w data blocks sorted in the fourth storage space.

10. The method for adjusting a storage position according to claim 1, wherein: After adjusting the storage positions of the data blocks allocated in the different storage spaces according to the first access frequencies corresponding to the access operations, the method further includes: flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage; A new second log file corresponding to the preset log mechanism is created, wherein the second log file is used to pre-write the read and write operations corresponding to the data blocks stored in the fifth storage space within the next time interval, and the fifth storage space is the first storage space after the storage position of the stored data blocks is adjusted.

11. The method for adjusting the storage position according to claim 10, wherein: Before flushing the data recorded in the first log file corresponding to the preset log mechanism to the permanent storage, the method further includes: In the event of an abnormal event, the data blocks in the first storage space that have not been written to the permanent storage are restored to the last consistent state through the replay mechanism corresponding to the first log file.

12. A storage position adjustment device, characterized in that: include: an allocation module, configured to allocate a group of data blocks to different storage spaces pre-divided in the cache according to a preset policy, wherein the different storage spaces include: a first storage space that allows execution of a preset log mechanism and a second storage space that allows synchronization updates with permanent storage, and the preset policy is used to instruct data blocks in the same access frequency range to be allocated to the same storage space; The adjustment module is used to receive access operations to the data blocks allocated to the different storage spaces at preset time intervals, and adjust the storage positions of the data blocks allocated to the different storage spaces in the different storage spaces according to the first access frequencies corresponding to the access operations.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the storage position adjustment method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the storage position adjustment method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the storage position adjustment method according to any one of claims 1 to 11 are implemented.

Images