Data storage control methods and devices, storage media and electronic equipment

Abstract

This application discloses a data storage control method and apparatus, storage medium, and electronic device, relating to the field of computer technology. The method includes: predicting the access frequency of multiple data points within a future time period based on historical access characteristics of these data points in a storage system; detecting whether the data access type of the multiple data points has changed based on the frequency range within which the access frequency falls; adjusting the storage strategy of the target data according to the changed target data access type when a change in the target data's data access type is detected, thereby obtaining a target storage strategy; and performing a target change operation on the target data, wherein the target change operation is used to change the target data's storage strategy to the target storage strategy. This solves the technical problem that storage systems struggle to balance storage overhead and storage performance, achieving the technical effect of balancing storage overhead and storage performance in data storage systems.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a data storage control method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] Traditional data storage strategies mainly include replica storage and erasure coding storage. Replica storage strategies (such as triple replica) ensure data reliability and high access performance by replicating data on multiple nodes. However, this approach generates huge redundant storage overhead, especially when cold data (such as archived logs and historical records) accounts for more than 70% of the total storage. The storage cost is very high, and it is impossible to achieve an effective balance between access performance and storage cost.

[0003] Erasure coding storage (such as Reed-Solomon coding) aims to address the high storage overhead of replica storage. It effectively reduces storage overhead by splitting data into multiple blocks and generating parity blocks for storage. For example, a 10+4 erasure coding ratio can reduce storage overhead by 50% compared to a traditional three-replica strategy. However, this strategy has shortcomings in storage performance. Since data recovery requires reading multiple blocks, access latency for cold data is high. Furthermore, existing erasure coding strategies are mostly statically configured, resulting in redundant storage of cold data, further impacting storage overhead optimization. Therefore, existing technologies struggle to achieve the optimal balance between storage overhead and storage performance.

[0004] There is still no effective solution to the technical problem that storage systems struggle to balance storage overhead and storage performance in related technologies. Summary of the Invention

[0005] This application provides a data storage control method and apparatus, storage medium and electronic device, to at least solve the technical problem in the related art that storage systems are unable to balance storage overhead and storage performance when storing data.

[0006] According to one embodiment of the present application, a data storage control method is provided, including:

[0007] Based on the historical access characteristics of multiple data in the storage system, predict the access frequency of multiple data in a future time period, wherein the historical access characteristics are used to indicate the access status of the corresponding data in the historical time period.

[0008] Based on the frequency range that the access frequency falls into, it is detected whether the data access type of multiple data has changed, wherein the data access type is used to indicate the access probability of the corresponding data being accessed in the future time period;

[0009] When a change in the data access type of the target data is detected, the storage strategy of the target data is adjusted according to the changed target data access type to obtain a target storage strategy. The storage overhead and storage performance of the data using different storage strategies are different. The storage overhead is used to indicate the system resources of the storage system required to store the corresponding data, and the storage performance is used to indicate the access speed when the corresponding data is accessed. The higher the access probability indicated by the target data access type, the higher the storage performance corresponding to the target storage strategy. The lower the access probability indicated by the target data access type, the lower the storage overhead corresponding to the target storage strategy.

[0010] Perform a target change operation on the target data, wherein the target change operation is used to change the storage policy of the target data to the target storage policy.

[0011] According to another embodiment of the present application, a data storage control device is also provided, comprising:

[0012] The prediction module is used to predict the access frequency of multiple data in a future time period based on the historical access characteristics of multiple data in the storage system, wherein the historical access characteristics are used to indicate the access status of the corresponding data in the historical time period.

[0013] The detection module is used to detect whether the data access type of multiple data has changed according to the frequency range in which the access frequency falls, wherein the data access type is used to indicate the access probability of the corresponding data being accessed in the future time period.

[0014] An adjustment module is used to adjust the storage strategy of the target data according to the changed target data access type when a change in the data access type of the target data is detected, thereby obtaining a target storage strategy. The storage overhead and storage performance of the data using different storage strategies are different. The storage overhead indicates the system resources of the storage system required to store the corresponding data, and the storage performance indicates the access speed when the corresponding data is accessed. The higher the access probability indicated by the target data access type, the higher the storage performance corresponding to the target storage strategy; the lower the access probability indicated by the target data access type, the lower the storage overhead corresponding to the target storage strategy.

[0015] An execution module is used to perform a target change operation on the target data, wherein the target change operation is used to change the storage policy of the target data to the target storage policy.

[0016] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the control method for data storage described above when executing the computer program.

[0017] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described data storage control methods.

[0018] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-described data storage control methods.

[0019] This application provides a data storage control method aimed at solving the problem of balancing storage overhead and storage performance in existing technologies. The method includes predicting the access frequency of multiple data items in a future time period based on their historical access characteristics. Then, based on the frequency range falling within which the access frequencies fall, it detects whether the data access type of the multiple data items has changed, where the data access type indicates the probability of the data being accessed in the future time period. When a change in the data access type of target data is detected, this method adjusts the storage strategy of the target data according to the changed target data access type to obtain a target storage strategy. Data using different storage strategies corresponds to different storage overhead and storage performance. The key to this method's ability to solve the problems of existing technologies lies in its dynamic correlation between data access probability and storage overhead and storage performance. Specifically, for target data with a higher access probability, the target storage strategy corresponds to higher storage performance to ensure fast access. Conversely, for target data with a lower access probability, the target storage strategy corresponds to lower storage overhead, thereby optimizing storage resources. Finally, a target change operation is performed on the target data to change the storage strategy to the target storage strategy. Through this dynamic and adaptive mechanism, this solution effectively balances storage overhead and storage performance, resolving the core contradiction in existing technologies. Therefore, it can solve the technical problem in related technologies where storage systems struggle to balance storage overhead and storage performance when storing data, achieving the technical effect of balancing storage overhead and storage performance in storage systems. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a hardware structure block diagram of a computer device for a data storage control method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of a data storage control method according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a storage system according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram illustrating the number of data blocks and the number of check blocks in an erasure coding storage strategy according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a storage policy change process according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of a data access type confirmation process according to an embodiment of this application;

[0027] Figure 7 This is a structural block diagram of a data storage control device according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0030] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0031] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a computer device for a data storage control method according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data storage control method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0035] This embodiment provides a data storage control method. Figure 2 This is a flowchart of a data storage control method according to an embodiment of this application, such as... Figure 2As shown, the process includes the following steps:

[0036] Step S12: Predict the access frequency of multiple data in a future time period based on the historical access characteristics of multiple data in the storage system, wherein the historical access characteristics are used to indicate the access status of the corresponding data in the historical time period.

[0037] Step S14: Detect whether the data access type of multiple data has changed according to the frequency range into which the access frequency falls, wherein the data access type is used to indicate the access probability of the corresponding data being accessed in the future time period.

[0038] Step S16: When a change in the data access type of the target data is detected, the storage strategy of the target data is adjusted according to the changed target data access type to obtain a target storage strategy. The storage overhead and storage performance of the data using different storage strategies are different. The storage overhead is used to indicate the system resources of the storage system required to store the corresponding data, and the storage performance is used to indicate the access speed when the corresponding data is accessed. The higher the access probability indicated by the target data access type, the higher the storage performance corresponding to the target storage strategy. The lower the access probability indicated by the target data access type, the lower the storage overhead corresponding to the target storage strategy.

[0039] Step S18: Perform a target change operation on the target data, wherein the target change operation is used to change the storage strategy of the target data to the target storage strategy.

[0040] This embodiment provides a data storage control method aimed at solving the problem of balancing storage overhead and storage performance in existing technologies. The method includes predicting the access frequency of multiple data items in a future time period based on their historical access characteristics. Then, based on the frequency range falling within which the access frequencies fall, it detects whether the data access types of the multiple data items have changed, where the data access type indicates the probability of the data being accessed in the future time period. When a change in the data access type of target data is detected, this method adjusts the storage strategy of the target data according to the changed target data access type to obtain a target storage strategy. Data using different storage strategies corresponds to different storage overhead and storage performance. The key to this method's ability to solve the problems of existing technologies lies in its dynamic correlation between data access probability and storage overhead and storage performance. Specifically, for target data with a higher access probability, the target storage strategy corresponds to higher storage performance to ensure fast access. Conversely, for target data with a lower access probability, the target storage strategy corresponds to lower storage overhead, thereby optimizing storage resources. Finally, a target change operation is performed on the target data to change the storage strategy to the target storage strategy. Through this dynamic and adaptive mechanism, this solution effectively balances storage overhead and storage performance, resolves the core contradiction in existing technologies, and achieves the technical effect of balancing storage overhead and storage performance in data storage systems. This solves the technical problem that storage systems struggle to balance storage overhead and storage performance in data storage.

[0041] Optionally, in this embodiment, the storage system may be, but is not limited to, a dynamic adaptive erasure coding (DA-EC) system. Figure 3 This is a schematic diagram of a storage system according to an embodiment of this application, such as... Figure 3 As shown, the architecture of the Dynamic Adaptive Erasure Coding (DA-EC) system (storage system) can be divided into four modules: data input layer, control layer, execution layer, and metadata management layer.

[0042] As the system's entry point, the data input layer is responsible for receiving and processing various data read and write requests from users, ensuring that users can interact with the system in an intuitive and efficient manner.

[0043] The control layer is the system's decision-making center, comprising a data classification module and a dynamic switching engine. The data classification module predicts the access frequency of data in the future based on its historical access characteristics. The dynamic switching engine, based on the frequency range falling within the predicted access frequency, detects whether the data access type has changed and intelligently determines the data storage strategy based on the data classification results and the current system load. When a change in the data access type of target data is detected, the dynamic switching engine adjusts the target data's storage strategy according to the changed data access type, resulting in the target storage strategy. The data access type indicates the probability of data being accessed in the future.

[0044] The execution layer is the actual storage operation part, including multi-replica storage groups and erasure coding storage groups. Multi-replica storage groups provide low-latency, highly available storage services for frequently accessed (hot) data. Erasure coding storage groups are suitable for less frequently accessed (warm) and cold data, using erasure coding technology to optimize storage space utilization while ensuring data reliability. Data using different storage strategies has different storage overhead and performance characteristics.

[0045] The metadata management layer is implemented using the distributed database Etcd, which is responsible for storing and managing the system's metadata information. Metadata includes key information such as data classification, storage location, number of replicas, and erasure coding parameters, supporting the efficient operation of the system.

[0046] As an optional approach, adjusting the storage strategy of the target data according to the target data access type after the target data change to obtain the target storage strategy further includes:

[0047] S21, if the access probability indicated by the target data access type is greater than or equal to a preset probability threshold, the storage strategy of the target data is adjusted to a replica storage strategy, wherein the target storage strategy includes the replica storage strategy, which is a strategy of storing the target data as one or more replicas.

[0048] S22, if the access probability indicated by the target data access type is less than the preset probability threshold, the storage strategy of the target data is adjusted to an erasure coding storage strategy, wherein the target storage strategy includes the erasure coding storage strategy, which is a strategy of dividing the target data into multiple data blocks and generating multiple check blocks for multiple data blocks for storage.

[0049] The storage overhead of the data using the replica storage strategy is higher than that of the data using the erasure coding storage strategy, and the storage performance of the data using the replica storage strategy is higher than that of the data using the erasure coding storage strategy.

[0050] Optionally, in this embodiment, when the access probability indicated by the target data access type is greater than or equal to a preset probability threshold, i.e., the target data access type is hot data, the storage system adopts a multi-replica storage method (e.g., 2 replicas) and stores it on low-latency SSD media nodes to provide high availability and low latency services. Conversely, when the access probability indicated by the target data access type is less than the preset probability threshold, i.e., the target data access type is cold data, the storage strategy will be adjusted to an erasure coding storage strategy. Erasure coding storage optimizes storage space utilization by dividing data into multiple data blocks and generating check blocks. The storage overhead of the replica storage strategy (e.g., 2 replicas) is higher than that of erasure coding storage (e.g., a 14+2 ratio, with an overhead of 1.14x), but its access performance is also higher.

[0051] As an optional solution, adjusting the storage strategy of the target data to an erasure coding storage strategy further includes:

[0052] S31, detect key parameters of the target data, wherein the key parameters are used to indicate the criticality of the target data in the storage system;

[0053] S32, determine the number of data blocks and the number of check blocks corresponding to the target data based on the key parameters and the access frequency predicted by the target data, and generate a target erasure coding storage strategy for the target data based on the number of data blocks and the number of check blocks. The target erasure coding storage strategy is a strategy of dividing the target data into the number of data blocks and generating the number of check blocks for the number of data blocks and storing them.

[0054] S33, the target erasure coding storage strategy is determined as the erasure coding storage strategy.

[0055] Optionally, in this embodiment, identifying the criticality of data (such as whether it is sensitive data, core business data, etc.) provides a basis for subsequent coding strategy differentiation. Data can be divided into two categories: "critical data" and "ordinary data," with key parameters being the core indicators used to distinguish between the two (such as data sensitivity, business importance, etc.). For example, metadata and core user information are marked as critical data, while ordinary logs and non-sensitive historical records are marked as ordinary data. This classification directly corresponds to the "detection of key parameters" in step S31, ensuring that the system can identify the criticality of the target data.

[0056] Furthermore, by combining the criticality of the data (key parameters) and the predicted access frequency, the number of data blocks and check blocks is dynamically allocated to balance fault tolerance and storage overhead. Figure 4 This is a schematic diagram illustrating the number of data blocks and the number of check blocks in an erasure coding storage strategy according to an embodiment of this application, as shown below. Figure 4 As shown, critical data requires higher fault tolerance, therefore more check blocks are allocated; ordinary data can reduce fault tolerance redundancy and decrease the number of check blocks. Critical data: uses "10+6RS encoding", i.e., 10 data blocks + 6 check blocks (tolerating 6 node failures); Ordinary data: further subdivided according to access frequency (warm / cold data): Warm data (access frequency 10-100 times / day): uses "12+4RS encoding", 12 data blocks + 4 check blocks (tolerating 4 node failures); Cold data (access frequency <10 times / day): uses "14+2RS encoding", 14 data blocks + 2 check blocks (tolerating 2 node failures).

[0057] The lower the access frequency (cold data), the lower the real-time performance requirement. Storage overhead can be reduced by increasing the number of data blocks and decreasing the number of parity blocks (cold data storage overhead is 1.14 times, lower than the 1.33 times of warm data). For medium access frequency (warm data), performance and overhead need to be balanced, and a medium number of parity blocks should be used. For example, if the target data is "critical data and accessed 5 times / day (cold data)," the basic number of parity blocks is determined based on the critical parameters (critical data), and the number of data blocks is adjusted according to the access frequency (cold data), ultimately using "10+6RS encoding"; if it is "ordinary data and accessed 50 times / day (warm data)," then "12+4RS encoding" is used.

[0058] After deciding on the storage strategy, the "Dynamic Switching Engine" records the target erasure coding parameters (number of data blocks / parity blocks, storage node location, etc.) through the metadata management layer (Etcd) and uses them as operation instructions for the execution layer. For example, for cold data that is determined to use "14+2RS encoding", the system writes the parameter "14 data blocks and 2 parity blocks" into the metadata. The execution layer divides the file into 1MB data blocks (D1-D14) according to this rule, generates 2 parity blocks (P1-P2), and distributes them across nodes in a cross-rack system, thus completing the implementation of the erasure coding storage strategy.

[0059] This solution proposes an adaptive erasure coding strategy module that classifies data based on data tags (cold data / warm data / critical data / hot data) and implements a specific block coding strategy for each data type to achieve an optimal balance between storage efficiency and fault tolerance. Table 1 details the number of blocks, check blocks, fault tolerance, and storage overhead for each data type. These parameters are designed based on data characteristics and system requirements to ensure that storage resource utilization is optimized to the maximum extent while meeting data reliability and availability needs. Through this flexible coding strategy, the system can dynamically adjust the storage method according to different data characteristics and requirements, thereby improving storage efficiency and system performance while ensuring data security. A seamless migration guarantee mechanism encompasses temporary cache design and conflict resolution strategies. During data migration, read requests can preferentially access temporary copies, while write requests are synchronized to both old and new storage locations, ensuring business continuity. Simultaneously, distributed locks (such as Redis) are used to properly resolve data consistency conflicts during the migration process.

[0060] Table 1 Fault Tolerance Matrix

[0061]

[0062] Through the complete process of "classification of critical data and ordinary data (S31), dynamic coding parameter allocation based on type and access frequency (S32), and metadata recording and execution layer landing (S33)" in this embodiment, the core is to achieve a precise balance between fault tolerance (1-6 node failures) and storage overhead (1.14-1.60 times) through differentiated erasure coding strategy, so as to ensure the high reliability of critical data and reduce the storage cost of ordinary low-frequency data.

[0063] As an optional solution, when the storage strategy of the target data is adjusted to the replica storage strategy, the step of performing the target change operation on the target data further includes:

[0064] S41, the target data is set to a locked state, wherein the target data in the locked state is only allowed to be modified by one process at any given time.

[0065] S42, the target data is copied to obtain a temporary data copy, and the temporary data copy is stored in the temporary copy storage node of the storage system, wherein the response speed of the data in the temporary copy storage node to the read request is greater than a preset first speed threshold.

[0066] S43, the access address of the target data in the storage system is changed to the first access address of the temporary data copy in the temporary copy storage node. At the same time, a target process of the storage system is called to convert the target data from erasure coding storage format to copy storage format to obtain a formal data copy. The formal data copy is then stored in the formal copy storage node of the storage system. The data in the storage system belonging to the erasure coding storage format is divided into multiple data blocks, and multiple check blocks are generated for multiple data blocks for storage. The data belonging to the copy storage format is copied into one or more copies for storage. The response speed of the data in the formal copy storage node to the read request is greater than a preset second speed threshold.

[0067] S44, if the formal data copy has already been stored on the formal copy storage node, change the access address of the target data in the storage system to the second access address of the formal data copy in the formal copy storage node, and perform a deletion operation on the target data stored in the storage system according to the erasure coding storage format and the temporary data copy.

[0068] Optionally, in this embodiment, Figure 5 This is a schematic diagram of a storage policy change process according to an embodiment of this application, such as... Figure 5 As shown, suppose in a storage system, a piece of e-commerce order data was originally classified as cold data due to its low access frequency (8 times / day) and stored using "14+2RS erasure coding" (14 data blocks + 2 parity blocks, stored on cross-rack mechanical hard drive nodes). Based on a 3-layer LSTM model prediction, the access frequency of this data will increase to 130 times / day in the next 7 days (due to an upcoming promotional activity), and the current cluster load rate is 50% (resources are sufficient, and the hot data threshold has dropped to 80 times / day). Therefore, the data access type needs to be changed from "cold data" to "hot data," requiring a strategy change from erasure coding to multi-replica storage, including the following steps:

[0069] 1. The storage system executes the command redis.setex('migration_lock:order_data',10, 'locked') to lock the order data resource for 10 seconds, preventing multiple processes from modifying the data simultaneously and ensuring that only one process is allowed to perform the migration operation at any given time.

[0070] 2. The system adopts COW (Copy-on-Write) technology, which only copies the accessed order data blocks (instead of the full data) to reduce redundant operations; the temporary copy (i.e., the temporary data copy) is stored on the SSD node (low-latency medium) in the same rack as the original erasure coding storage (i.e., the temporary copy storage node), ensuring that the read request response speed is <5ms (meeting the requirement that "the response speed is greater than the preset first speed threshold").

[0071] 3. The metadata management layer (Etcd) changes the access address of order data from the original erasure coding block address to the first access address of the temporary replica on the SSD node, and user read and write requests are preferentially routed to the temporary replica; the background thread starts the format conversion, converting the order data from "14+2 erasure coding format" to "2 replica storage format", and stores the official replica on the high-performance SSD official storage node (response speed greater than the preset second speed threshold); during the migration process, write requests are simultaneously synchronized to the temporary replica and official replica nodes to ensure data consistency.

[0072] 4. Once the formal replica storage node confirms the storage is complete, the metadata access address will be changed from the first access address of the temporary replica to the second access address of the formal replica. The system will automatically delete the original erasure coding block (14 data blocks + 2 check blocks) and the temporary replica, releasing redundant storage resources and completing the closed loop of the policy change from erasure coding to multiple replicas.

[0073] As seen in the above order data migration scenario, the complete migration process—"distributed locks ensuring atomicity → rapid response from COW temporary replicas → asynchronous migration and address updates → old data cleanup"—is achieved. The core principle is to convert erasure coding storage for low-frequency data to multi-replica storage for high-frequency data without interrupting business operations, balancing storage performance and data consistency. During the data migration process, the dynamic switching engine uses COW (Copy-on-Write) technology to create temporary replicas, copying only the accessed blocks to reduce redundant operations and improve efficiency. The temporary replicas are stored on SSD nodes in the same rack as the original erasure coding data, ensuring both rapid response and data security. To guarantee the atomicity of the migration, the system employs a distributed lock (such as Redis) mechanism to lock resources during critical migration phases. This ensures that read requests prioritize access to the temporary replicas, and write requests are synchronized to both the old and new storage locations, preventing data conflicts and ensuring data consistency and integrity. A pseudocode example of an atomic migration based on distributed locks is as follows:

[0074] lock:= redis.Lock("migration_lock:"+data_id, 10*time.Second)

[0075] defer lock.Release()

[0076] if lock.Acquired() {

[0077] create_temp_copy()

[0078] update_metadata()

[0079] async_migrate()

[0080] }

[0081] As an optional solution, when the storage strategy of the target data is adjusted to the erasure coding storage strategy, the step of performing a target change operation on the target data further includes:

[0082] S51, the target data is converted from the replica storage format to the erasure coding storage format to obtain multiple target data blocks and multiple target verification blocks of the target data. In the storage system, the data belonging to the erasure coding storage format is divided into multiple data blocks, and multiple verification blocks are generated for the multiple data blocks for storage. The data belonging to the replica storage format is copied into one or more replicas for storage.

[0083] S52, the multiple target data blocks and the multiple target verification blocks are stored in the multiple erasure coding storage nodes of the storage system, wherein the storage overhead required to store the data through the multiple erasure coding storage nodes is lower than a preset overhead threshold;

[0084] S53, change the access address of the target data in the storage system to a third access address, wherein the third access address is the address required to read multiple target data blocks and multiple target check blocks from multiple erasure coding storage nodes;

[0085] S54, perform a deletion operation on the target data stored in the storage system according to the replica storage format.

[0086] Optionally, in this embodiment, it is assumed that in the storage system, a set of game user login data is classified as hot data due to frequent access during a recent event (150 times / day), and a "2-replica SSD storage" strategy is adopted (two replicas are stored on SSD nodes in different racks). After the event ends, according to the prediction of a 3-layer LSTM model, the access frequency of this data will drop to 15 times / day in the next 7 days, and the current cluster load rate is 85% (resources are tight, and the cold data threshold drops to 8 times / day). Therefore, the data access type changes from "hot data" to "warm data", and a strategy change from multi-replica to erasure coding storage needs to be performed, including the following steps:

[0087] 1. The system reads the game login data from the original two-copy storage and divides it into 12 data blocks (D1-D12) of 1MB each. Since this data is ordinary business data (non-critical data) and the access type is warm data, four verification blocks (P1-P4) are generated according to the rules, completing the conversion from "multi-copy format" to "12+4RS erasure coding format" (corresponding to...). Figure 4 “Warm Data 12+4RS Encoding Process”.

[0088] 2. The system evenly distributes 12 data blocks and 4 parity blocks to mechanical hard disk nodes (non-SSD nodes) in 4 different racks to reduce the performance overhead of high-frequency access; the overhead of this storage method is 1.33 times that of the original data (16 blocks / 12 data blocks), which is significantly lower than the 2.0 times overhead of replica storage, meeting the requirement of "storage overhead is lower than the preset threshold".

[0089] 3. The metadata database (Etcd) updates the storage information of the game login data, changing the original access address pointing to the SSD replica node to a "third access address" containing 16 block storage locations (i.e., the distributed addresses for reading D1-D12 and P1-P4 from 4 rack nodes); after the update, user read requests will be routed to the corresponding erasure coding node through the third access address, and the complete data will be restored through the combination of data blocks and check blocks.

[0090] 4. After confirming that erasure coding block storage is complete and the access address is updated, the system starts a background cleanup thread to delete the two replicas of data on the original SSD node. After the cleanup is complete, only the 16 blocks of erasure coding storage are retained, and SSD resources are released for high-frequency access data to achieve optimized allocation of storage resources.

[0091] As seen in the above game login data migration scenario, the entire process of "format conversion (block encoding) → distributed storage (cross-rack nodes) → address update (metadata synchronization) → old copy cleanup" is fully implemented. The core is to convert high-overhead copy storage into low-overhead erasure coding storage after the data access frequency decreases, thereby reducing system storage costs while ensuring data reliability.

[0092] As an optional solution, the step of detecting whether the data access type of multiple data items has changed based on the frequency range falling within the access frequency further includes:

[0093] The following steps are used to detect whether the data access type of the reference data has changed, wherein the reference data is any data in the storage system:

[0094] S61, obtain the target correspondence between the multiple frequency ranges and the multiple data access types;

[0095] S62, Match the reference data access type corresponding to the reference frequency range into which the reference access frequency of the reference data falls from the target correspondence;

[0096] S63, Detect whether the reference data access type is consistent with the current data access type of the reference data;

[0097] S64, if it is detected that the access type of the reference data is inconsistent with the current access type of the reference data, it is determined that the access type of the reference data has changed.

[0098] Optionally, in this embodiment, Figure 6 This is a schematic diagram of a data access type confirmation process according to an embodiment of this application, such as... Figure 6 As shown, step S61 uses the data classification module to receive data access logs and outputs predicted data access frequencies for the next 7 days using a trained 3-layer LSTM prediction model. Then, based on pre-set classification thresholds, the data is divided into three categories: hot data (>100 times / day), warm data (10-100 times / day), and cold data (<10 times / day). Here, "hot data," "warm data," and "cold data" are the "data access types," and the corresponding access frequency thresholds, such as ">100 times / day," "10-100 times / day," and "<10 times / day," are the "frequency ranges." The relationship between these classification thresholds and data types constitutes the "target correspondence."

[0099] Step S62 is performed based on the correspondence established in step S61. For any data in the storage system (i.e., "reference data"), its predicted access frequency (i.e., "reference access frequency") is first obtained. Then, this predicted frequency is compared with the frequency range mentioned above to determine which range it belongs to, and thus the corresponding "data access type" is matched. For example, if the predicted access frequency is 150 times / day, it will match a frequency range of ">100 times / day", thus determining its "reference data access type" as "hot data".

[0100] After obtaining the predicted "reference data access type," the system compares it with the current "data access type" of the data stored in the metadata management layer. If the prediction result does not match the current state, for example, the currently stored type is "cold data," while the predicted type is "hot data," it indicates that the data type has potentially changed.

[0101] When the comparison results are inconsistent, the storage system determines that the data access type has changed. This change triggers subsequent storage policy adjustments and migration operations. For example, if data changes from "cold data" to "hot data," the system will activate the dynamic switching engine to switch the data's storage policy from erasure coding to multi-replica storage to accommodate its high access frequency.

[0102] As an optional approach, obtaining the target correspondence between the multiple frequency ranges and the multiple data access types further includes:

[0103] S71, detect the current system load parameters of the storage system, wherein the system load parameters are used to indicate the load status of the system resources of the storage system;

[0104] S72, when the system load parameters fall within the first load range, a first correspondence between the multiple frequency ranges and the multiple data access types is determined as the target correspondence, wherein the first load range is a load range that is greater than or equal to the target load lower limit and less than or equal to the target load upper limit, and the first correspondence is: the first frequency range corresponds to the hot access type, the second frequency range corresponds to the cold access type, the multiple frequency ranges include the first frequency range and the second frequency range, the first frequency range is a frequency range whose lower limit is a preset target frequency lower limit, the second frequency range is a frequency range whose upper limit is a preset target frequency upper limit, the multiple data access types include the hot access type and the cold access type, the access probability indicated by the hot access type is greater than or equal to a preset probability threshold, and the access probability indicated by the cold access type is less than the preset probability threshold;

[0105] S73, when the system load parameters fall into the second load range, the lower limit of the target frequency in the first frequency range in the first correspondence is lowered by a first frequency threshold, and the upper limit of the target frequency in the second frequency range is increased by a second frequency threshold to obtain a second correspondence, and the second correspondence is determined as the target correspondence, wherein the second load range is a load range that is less than the lower limit of the target load.

[0106] S74, when the system load parameters fall into the third load range, the lower limit of the target frequency in the first frequency range in the first correspondence is raised to a third frequency threshold, and the upper limit of the target frequency in the second frequency range is lowered to a fourth frequency threshold to obtain a third correspondence. The third correspondence is determined as the target correspondence, wherein the third load range is a load range greater than the upper limit of the target load.

[0107] Optionally, in this embodiment, it is assumed that a storage system is operating normally with a cluster load rate of 75%, which is within the first load range (60%-80%). At this time, the system adopts a first correspondence: the access frequency threshold for hot data is set to 100 times / day, and the access frequency threshold for cold data is set to 10 times / day.

[0108] When a sudden surge in system load causes the cluster load to reach 85%, the system load parameters fall into the third load range (>80%). At this point, the system initiates elastic threshold adjustment, raising the hot data threshold to 120 times / day to reduce the scale of hot data and ensure the performance of frequently accessed data. Simultaneously, the cold data threshold is lowered to 8 times / day, allowing more low-frequency data to be stored in cold data, thereby optimizing storage resource allocation and alleviating cluster load pressure. Through this adjustment, data originally accessed 100-120 times / day will no longer be considered hot data, thus reducing the demand for high-performance storage resources. Ultimately, this strategy successfully reduced the cluster load from 85% to 79%.

[0109] Conversely, if the system enters a low-load state, and the cluster load rate drops to 55% (falling into the second load range, <60%), the system will make the opposite adjustment: the hot data threshold will be lowered from 100 times / day to 80 times / day, allowing more data to benefit from the hot data storage strategy and improving access performance. Simultaneously, the cold data threshold will be raised from 10 times / day to 15 times / day, classifying some frequently accessed data as warm data and adopting a warm data storage strategy to make rational use of storage space. This approach aims to fully utilize idle storage resources and improve overall access performance.

[0110] This solution dynamically adjusts data thresholds based on cluster load to optimize storage resource allocation. As shown in Table 2, when the cluster load is below 60%, storage resources are sufficient. The hot data access frequency threshold is reduced from 100 times / day to 80 times / day, allowing more data to benefit from the hot data storage strategy and improving access performance. Simultaneously, the cold data threshold is increased from 10 times / day to 15 times / day, classifying some frequently accessed data as warm data and adopting a warm data storage strategy to make reasonable use of storage space. If the cluster load is between 60% and 80%, storage pressure is moderate, and the hot and cold data thresholds remain unchanged, maintaining the existing data classification and storage strategy to ensure stable system operation. When the cluster load exceeds 80%, storage resources are strained. The hot data threshold is increased to 120 times / day to reduce the scale of hot data to ensure the performance of high-frequency data; the cold data threshold is reduced to 8 times / day, allowing low-frequency data to use cold data storage, optimizing storage resource allocation and alleviating cluster load pressure.

[0111] Table 2 Cluster Load Rate and Data Threshold Adjustment Strategies

[0112]

[0113] As an optional approach, the method of predicting the access frequency of multiple data items within a future time period based on their historical access characteristics in the storage system further includes:

[0114] The access frequency of the i-th data among N data in the storage system will be predicted within a future time period using the following steps, where N is an integer greater than or equal to 1, and i is a positive integer less than or equal to N:

[0115] S81, perform multiple rounds of sampling on the access information of the i-th data within the historical time period to obtain the i-th access feature sequence corresponding to the i-th data, wherein the access information includes records of the i-th data being accessed within the historical time period;

[0116] S82, input the i-th access feature sequence into the target prediction model to obtain the i-th access frequency output by the target prediction model, wherein the target prediction model is obtained by training the initial prediction model using access feature sequence samples labeled with access frequency tags;

[0117] S83, the i-th access frequency is determined as the access frequency of the i-th data.

[0118] Optionally, in this embodiment, it is assumed that there is a set of corporate financial statement data in the storage system, and it is necessary to predict its access frequency for the next 7 days. The historical access log of this data contains access records (timestamps, daily access frequency, data size, etc.) from the past 3 months, and the system needs to complete the prediction based on these historical features. The system adopts "hourly granular" sampling (i.e., recording access features once per hour), which meets the requirement of "access log data sampled at an hourly granular level based on the past 3 months". Five key features are extracted from the historical access log, including hourly access frequency, access duration, and data size ("the model input is five feature data points from the past 168 time points"). The hourly data from the past 3 months (approximately 90 days) is sampled, and the feature data of the most recent 168 time points (7 days × 24 hours) are taken and sorted in chronological order to form the "access feature sequence" of the financial statement data (e.g., [access frequency of 2 times in the 1st hour, access frequency of 0 times in the 2nd hour, ..., access frequency of 3 times in the 168th hour]).

[0119] The target prediction model can, but is not limited to, a 3-layer LSTM prediction model (128 hidden units). This LSTM model has been trained on a large amount of historical data. The input is "5 feature data points at 168 time points," and the output is "average daily access frequency for the next 7 days." The "access feature sequence" of the aforementioned financial statement data is input into the LSTM model. The model learns historical access patterns (e.g., higher access frequency at the beginning and end of each month, lower access frequency during the day) and outputs a prediction of the average daily access frequency for the next 7 days. For example, the predicted average daily access frequency for the next 7 days is 25 times / day. The system determines the "average daily access frequency of 25 times / day" output by the LSTM model as the future access frequency of the financial statement data, providing a basis for subsequent data access type classification (hot / warm / cold data). Based on the classification rules, 25 times / day falls into the "warm data (10-100 times / day)" range, triggering the corresponding erasure coding storage strategy (12+4RS encoding).

[0120] As can be seen from the prediction scenario based on the above financial statement data, the complete process of "generating feature sequences through hourly granular sampling → learning historical patterns through LSTM models → outputting access frequency for the next 7 days" is fully realized. The core is to use time series prediction models to mine the correlation between historical access features and future access patterns, providing accurate prediction basis for dynamic storage strategy adjustments.

[0121] As an optional approach, the step of sampling the access information of the i-th data within the historical time period in multiple rounds to obtain the i-th access feature sequence corresponding to the i-th data further includes:

[0122] S91, obtain the preset sampling time interval;

[0123] S92, sample the features of the access information of the i-th data within the historical time period according to the sampling time interval to obtain multiple historical access features;

[0124] S93, sort the multiple historical access features according to the sampling time order to obtain the i-th access feature sequence.

[0125] Optionally, in this embodiment, it is assumed that there is a user shopping record in the storage system that needs to predict future access frequency. The system needs to extract features from the historical access logs of this data over the past 3 months to generate an access feature sequence for input to the LSTM model. The system uses hourly granularity as the sampling time interval, that is, it collects features of the data's access information once per hour. This interval conforms to the explicit requirement of "training based on access log data sampled at hourly granularity over the past 3 months," ensuring that the access patterns of the data at different times of the day are captured (such as frequent access during the day and sparse access at night).

[0126] The system extracts key access features from the user's historical access logs of shopping records, at hourly intervals. These features include timestamps, access frequency, and data size; therefore, the specific historical access features obtained for each sample are as follows:

[0127] The number of visits within that hour (e.g., 10 times);

[0128] The average access time within that hour (e.g., 2 seconds / access);

[0129] The amount of data accessed within that hour (e.g., 5MB);

[0130] Access the source node (such as an SSD node in a server rack);

[0131] Access type (read / write operation ratio).

[0132] After continuous sampling, the system obtained hourly feature data for the past 3 months (approximately 90 days), collecting a total of 90×24=2160 historical access features.

[0133] Since the model input consists of "5 feature data points from the past 168 time points" (i.e., hourly data from the last 7 days), the system selects features from 2160 historical access features, focusing on the most recent 168 hours (7 days × 24 hours), and sorts them chronologically to form a structured access feature sequence. For example:

[0134] Characteristics of the first hour: [8 accesses, average duration 1.8 seconds, data volume 4MB, source node A, read operations account for 90%];

[0135] Characteristics of the second hour: [5 accesses, average duration 2.1 seconds, data volume 3MB, source node B, read operations account for 85%];

[0136] ...

[0137] Characteristics of the 168th hour: [12 accesses, average duration 2.3 seconds, data volume 6MB, source node A, read operations account for 95%].

[0138] The sequence, arranged chronologically, is used directly as input data for an LSTM model to predict the frequency of visits over the next 7 days.

[0139] As can be seen from the above sampling process of user shopping record data, the entire process of "hourly granular interval sampling → extraction of multi-dimensional access features → generation of sequences by sorting by time" is fully implemented. The core is to provide the LSTM model with structured input data that meets the training requirements through standardized time interval sampling and time sequence sorting, so as to ensure that the model can effectively learn historical access patterns.

[0140] The data storage control method proposed in this application introduces a dynamic data classification mechanism based on LSTM: A three-layer LSTM prediction model (with 128 hidden units in the hidden layer) is used as training data, employing hourly sampled access logs from the past three months. This model accurately predicts the access frequency for the next seven days, dynamically distinguishing between hot, cold, and warm data (thresholds are <60%, 60%-80%, and >80%, respectively), achieving real-time matching of data classification with storage resources. The model input consists of five feature data points from the past 168 time points, and the output is the average daily access frequency for the next seven days. Based on this result, data can be dynamically divided into hot data (>100 times / day), warm data (10-100 times / day), and cold data (<10 times / day), providing a precise basis for subsequent storage strategy adjustments.

[0141] A flexible threshold-driven storage strategy switching engine is also proposed: a dynamic switching engine is designed to execute differentiated storage strategies based on data classification results. Hot data storage adopts a multi-replica strategy (e.g., 2 replicas), stored on low-latency SSD nodes, and temporary replicas are created through COW technology. Redis distributed locks are used to ensure atomicity, enabling uninterrupted business switching. Warm / cold data storage uses an adaptive erasure coding ratio, with a 10+6 ratio for critical data and a 14+2 ratio for ordinary data. Checksum blocks are dynamically generated in 1MB blocks, and the migration process updates the status through the metadata database Etcd. The background asynchronously completes data migration and old block cleanup, improving storage efficiency.

[0142] Furthermore, a load-aware adaptive coding strategy is proposed, which dynamically adjusts coding parameters based on cluster load: under high load scenarios (>80%), the hot and cold data thresholds for warm data are reduced to 10⁻⁸ times / day to reduce redundant storage overhead; under low load scenarios (<60%), the hot data threshold is reduced to 80 times / day, and the cold data threshold is increased to 15 times / day to expand the coverage of high-performance storage. Through differentiated RS coding (2 copies for hot data, 12+4 for warm data, 14+2 for cold data, and 10+6 for critical data), an optimal balance is achieved between fault tolerance (1-6 node failures) and storage overhead (1.14-1.60 times), enabling fine-grained management of storage resources.

[0143] Furthermore, it features lossless migration and consistency assurance mechanisms: the migration process prioritizes temporary replicas (SSD nodes in the same rack) for responding to read requests, controlling latency to within 5ms, and synchronizing write operations to both old and new storage locations to ensure business continuity. Conflict resolution is based on distributed locks to ensure the atomicity of critical operations, such as `redis.setex('migration_lock', 10, 'locked')`, ensuring data consistency and business continuity. After migration, temporary replicas and old blocks are automatically deleted, reducing redundant storage overhead by up to 18% and improving storage resource utilization.

[0144] This application proposes a four-layer modular system: the data input layer processes user read and write requests, the control layer integrates a data classification module and a dynamic switching engine to determine storage strategies and executes parallel operations of multi-layer replica groups and erasure coding groups, and the metadata management layer (Etcd) uniformly manages key information such as data tags, locations, and encoding parameters, supporting high availability and fast access, and realizing efficient collaborative operation of the storage system.

[0145] The data storage control method proposed in this application enables automated optimization of storage performance. Regarding storage performance, the system implements automatic compression and cache optimization strategies. Automatic compression is initiated when the LSTM prediction model detects a high read / write frequency (reaching 160 time points / 5 features), employing different compression strategies: hot data (>150 times / day) uses multi-replica compression (3 replicas), achieving a compression time of 50ms and a read / write latency of 10ms; cold data (<60 times / day) utilizes high-density erasure coding (14+2 RS encoding), achieving a compression ratio of 1.4 times with a storage overhead of 1.14 times. The cache optimization strategy extends the caching time for hot data from 24 hours to 72 hours and shortens it to 24 hours for cold data. While increasing the cache hit rate to 90%, the cache eviction period is extended to 7 days.

[0146] Furthermore, decoupling optimization of the cluster and storage can be implemented to improve system performance and resource utilization, thereby reducing the migration of hot and cold data. By storing temporary replicas on the hot data SSD node (replicating only 20MB with the help of CDN technology) and using distributed locks (Redis) to ensure atomicity, the need for hot and cold data migration was successfully reduced by 40%. In addition, a load balancing mapping table was introduced to associate cluster load with data popularity and adjust accordingly: when the load is ≥60%, the hot data migration threshold is adjusted from 100 times / day to 15 times / day, and the cold data migration threshold is adjusted from 10 times / day to 15 times / day; when the load is ≥80%, the hot data migration threshold is further adjusted to 15 times / day, and the cold data migration threshold is 5 times / day. This strategy effectively reduced the load from 85% to 79% and improved cache utilization by 30%.

[0147] Furthermore, it features a dual protection mechanism, with the system constructing a dual guarantee mechanism for security and consistency. During data migration, synchronous read and write operations use temporary replicas (latency <5ms), and distributed locks ensure atomicity (e.g., the command `redis.setex('migration_lock:file123', 10, 'locked')`), ensuring uninterrupted business operations and a data loss probability of less than 0.1%. Regarding disaster recovery optimization, critical data is stored on four chassis nodes (e.g., metadata with two replicas + RS encoding distributed across four chassis), improving availability to withstand four-node failures.

[0148] Furthermore, the storage system possesses self-healing and resilience capabilities. To avoid manual, mechanical maintenance, the system is designed with a predictive automated maintenance mechanism. The LSTM model predicts performance bottlenecks for the next 7 days based on historical data (92% accuracy), triggering automatic migration with a success rate of up to 90%, reducing failover time to 1 minute. Simultaneously, it dynamically adjusts resilience: when the load is ≤60%, the cold data migration threshold is reduced to 50 times / day, allowing more data to receive high-performance storage and improving access performance by 40%.

[0149] Finally, in terms of economic benefits, the storage system achieved reduced storage costs and improved operational efficiency. With a 70% reduction in data activity, storage overhead decreased by 15% (through optimized storage strategies and replica counts), while operational efficiency improved by 60%: automated migration scripts (Ansible Playbook) reduced fault handling time by 60% (from 2 hours to 10 minutes). These optimizations significantly reduced operating costs while improving the overall efficiency and reliability of the system.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0151] Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0152] This embodiment also provides a data storage control device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0153] Figure 7 This is a structural block diagram of a data storage control device according to an embodiment of this application; as shown below. Figure 7 As shown, it includes:

[0154] The prediction module 702 is used to predict the access frequency of multiple data in a future time period based on the historical access characteristics of multiple data in the storage system, wherein the historical access characteristics are used to indicate the access status of the corresponding data in the historical time period.

[0155] The detection module 704 is used to detect whether the data access type of multiple data has changed according to the frequency range in which the access frequency falls, wherein the data access type is used to indicate the access probability of the corresponding data being accessed in the future time period.

[0156] The adjustment module 706 is used to adjust the storage strategy of the target data according to the changed target data access type when a change in the data access type of the target data is detected, thereby obtaining a target storage strategy. The storage overhead and storage performance of the data using different storage strategies are different. The storage overhead is used to indicate the system resources of the storage system required to store the corresponding data, and the storage performance is used to indicate the access speed when the corresponding data is accessed. The higher the access probability indicated by the target data access type, the higher the storage performance corresponding to the target storage strategy. The lower the access probability indicated by the target data access type, the lower the storage overhead corresponding to the target storage strategy.

[0157] The execution module 708 is used to perform a target change operation on the target data, wherein the target change operation is used to change the storage strategy of the target data to the target storage strategy.

[0158] In one exemplary embodiment, the adjustment module includes:

[0159] The first adjustment unit is configured to adjust the storage strategy of the target data to a replica storage strategy when the access probability indicated by the target data access type is greater than or equal to a preset probability threshold. The target storage strategy includes the replica storage strategy, which is a strategy of storing the target data as one or more replicas.

[0160] The second adjustment unit is used to adjust the storage strategy of the target data to an erasure coding storage strategy when the access probability indicated by the target data access type is less than the preset probability threshold. The target storage strategy includes the erasure coding storage strategy, which is a strategy of dividing the target data into multiple data blocks and generating multiple verification blocks for multiple data blocks for storage.

[0161] The storage overhead of the data using the replica storage strategy is higher than that of the data using the erasure coding storage strategy, and the storage performance of the data using the replica storage strategy is higher than that of the data using the erasure coding storage strategy.

[0162] In one exemplary embodiment, the second adjustment unit is further configured to:

[0163] Detect key parameters of the target data, wherein the key parameters are used to indicate the criticality of the target data in the storage system;

[0164] The number of data blocks and the number of check blocks corresponding to the target data are determined based on the key parameters and the access frequency predicted by the target data. A target erasure coding storage strategy is generated for the target data based on the number of data blocks and the number of check blocks. The target erasure coding storage strategy is a strategy of dividing the target data into the number of data blocks and generating and storing the number of check blocks for the number of data blocks.

[0165] The target erasure coding storage strategy is determined as the erasure coding storage strategy.

[0166] In an exemplary embodiment, when the storage policy of the target data is adjusted to the replica storage policy, the execution module includes:

[0167] The setting unit is used to set the target data to a locked state, wherein the target data in the locked state is only allowed to be modified by one process at any given time.

[0168] A copying unit is used to copy the target data to obtain a temporary data copy and store the temporary data copy in a temporary copy storage node in the storage system, wherein the response speed of the data in the temporary copy storage node to a read request is greater than a preset first speed threshold.

[0169] The first modification unit is used to change the access address of the target data in the storage system to the first access address of the temporary data copy in the temporary copy storage node, and simultaneously call a target process of the storage system to convert the target data from erasure coding storage format to copy storage format to obtain a formal data copy, and store the formal data copy in the formal copy storage node of the storage system. The data in the storage system belonging to the erasure coding storage format is divided into multiple data blocks, and multiple check blocks are generated for multiple data blocks for storage. The data belonging to the copy storage format is copied into one or more copies for storage. The response speed of the data in the formal copy storage node to the read request is greater than a preset second speed threshold.

[0170] The first deletion unit is configured to, when the formal data copy has already been stored on the formal copy storage node, change the access address of the target data in the storage system to the second access address of the formal data copy in the formal copy storage node, and perform a deletion operation on the target data stored in the storage system according to the erasure coding storage format, as well as the temporary data copy.

[0171] In an exemplary embodiment, when the storage strategy of the target data is adjusted to the erasure coding storage strategy, the execution module includes:

[0172] The conversion unit is used to convert the target data from the replica storage format to the erasure coding storage format to obtain multiple target data blocks and multiple target verification blocks of the target data. In the storage system, the data belonging to the erasure coding storage format is divided into multiple data blocks, and multiple verification blocks are generated for multiple data blocks for storage. The data belonging to the replica storage format is copied into one or more replicas for storage.

[0173] A storage unit is used to store multiple target data blocks and multiple target verification blocks to multiple erasure coding storage nodes of the storage system, wherein the storage overhead required to store the data through the multiple erasure coding storage nodes is lower than a preset overhead threshold.

[0174] The second modification unit is used to change the access address of the target data in the storage system to a third access address, wherein the third access address is the address required to read multiple target data blocks and multiple target check blocks from multiple erasure coding storage nodes;

[0175] The second deletion unit is used to perform a deletion operation on the target data stored in the storage system according to the copy storage format.

[0176] In one exemplary embodiment, the detection module includes:

[0177] The detection unit is configured to detect whether the data access type of the reference data has changed through the following steps, wherein the reference data is any data in the storage system:

[0178] Obtain the target correspondence between multiple frequency ranges and multiple data access types;

[0179] Match the reference data access type corresponding to the reference frequency range into which the reference access frequency of the reference data falls from the target correspondence;

[0180] Detect whether the access type of the reference data is consistent with the current access type of the reference data;

[0181] If the access type of the reference data is found to be inconsistent with the current access type of the reference data, it is determined that the access type of the reference data has changed.

[0182] In one exemplary embodiment, the detection unit is further configured to:

[0183] The system load parameters of the storage system are detected, wherein the system load parameters are used to indicate the load status of the system resources of the storage system;

[0184] When the system load parameters fall within a first load range, a first correspondence between multiple frequency ranges and multiple data access types is determined as the target correspondence. The first load range is a load range greater than or equal to a target lower load limit and less than or equal to a target upper load limit. The first correspondence is as follows: a first frequency range corresponds to a hot access type, and a second frequency range corresponds to a cold access type. The multiple frequency ranges include the first frequency range and the second frequency range. The first frequency range is a frequency range with a lower limit equal to a preset target lower frequency limit, and the second frequency range is a frequency range with an upper limit equal to a preset target upper frequency limit. The multiple data access types include the hot access type and the cold access type. The access probability indicated by the hot access type is greater than or equal to a preset probability threshold, and the access probability indicated by the cold access type is less than the preset probability threshold.

[0185] When the system load parameters fall into the second load range, the lower limit of the target frequency in the first frequency range in the first correspondence is lowered by a first frequency threshold, and the upper limit of the target frequency in the second frequency range is increased by a second frequency threshold to obtain a second correspondence. The second correspondence is determined as the target correspondence, wherein the second load range is a load range that is less than the lower limit of the target load.

[0186] When the system load parameters fall into the third load range, the lower limit of the target frequency in the first frequency range in the first correspondence is raised to a third frequency threshold, and the upper limit of the target frequency in the second frequency range is lowered to a fourth frequency threshold to obtain the third correspondence. The third correspondence is determined as the target correspondence, wherein the third load range is a load range greater than the upper limit of the target load.

[0187] In one exemplary embodiment, the prediction module includes:

[0188] The prediction unit is configured to predict the access frequency of the i-th data among N data in the storage system within a future time period through the following steps: where N is an integer greater than or equal to 1, and i is a positive integer less than or equal to N.

[0189] Multiple rounds of sampling are performed on the access information of the i-th data within the historical time period to obtain the i-th access feature sequence corresponding to the i-th data, wherein the access information includes records of the i-th data being accessed within the historical time period;

[0190] The i-th access feature sequence is input into the target prediction model to obtain the i-th access frequency output by the target prediction model, wherein the target prediction model is obtained by training the initial prediction model using access feature sequence samples labeled with access frequency tags;

[0191] The i-th access frequency is determined as the access frequency of the i-th data.

[0192] In one exemplary embodiment, the prediction unit is further configured to:

[0193] Obtain the preset sampling time interval;

[0194] The features of the access information of the i-th data within the historical time period are sampled according to the sampling time interval to obtain multiple historical access features;

[0195] The historical access features are sorted according to the sampling time order to obtain the i-th access feature sequence.

[0196] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0197] For a description of the features in the embodiment corresponding to the data storage control device, please refer to the relevant description in the embodiment corresponding to the data storage control method, which will not be repeated here.

[0198] Embodiments of this application also provide an electronic device. Figure 8 This is a schematic diagram of an electronic device according to an embodiment of this application, such as... Figure 8 As shown, the electronic device includes a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above-described data storage control method embodiments.

[0199] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0200] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0201] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data storage control method embodiments when it runs.

[0202] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0203] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0204] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application; the computer program product further includes a non-volatile computer-readable storage medium storing the computer program, which, when executed by a processor, implements the steps of the data storage control method described in various embodiments of this application.

[0205] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0206] The data storage control method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data storage control method, characterized in that, include: Based on the historical access characteristics of multiple data in the storage system, predict the access frequency of multiple data in a future time period, wherein the historical access characteristics are used to indicate the access status of the corresponding data in the historical time period. Based on the frequency range that the access frequency falls into, it is detected whether the data access type of multiple data has changed, wherein the data access type is used to indicate the access probability of the corresponding data being accessed in the future time period; When a change in the data access type of the target data is detected, the storage strategy of the target data is adjusted according to the changed target data access type to obtain a target storage strategy. The storage overhead and storage performance of the data using different storage strategies are different. The storage overhead is used to indicate the system resources of the storage system required to store the corresponding data, and the storage performance is used to indicate the access speed when the corresponding data is accessed. The higher the access probability indicated by the target data access type, the higher the storage performance corresponding to the target storage strategy. The lower the access probability indicated by the target data access type, the lower the storage overhead corresponding to the target storage strategy. Perform a target change operation on the target data, wherein the target change operation is used to change the storage policy of the target data to the target storage policy; The step of adjusting the storage strategy of the target data according to the target data access type after the target data change to obtain a target storage strategy includes: adjusting the storage strategy of the target data to a replica storage strategy when the access probability indicated by the target data access type is greater than or equal to a preset probability threshold, wherein the target storage strategy includes the replica storage strategy, which is a strategy of storing the target data as one or more replicas; and adjusting the storage strategy of the target data to an erasure coding storage strategy when the access probability indicated by the target data access type is less than the preset probability threshold, wherein the target storage strategy includes the erasure coding storage strategy, which is a strategy of dividing the target data into multiple data blocks and generating multiple check blocks for multiple data blocks for storage; wherein the storage overhead corresponding to the data using the replica storage strategy is higher than the storage overhead corresponding to the data using the erasure coding storage strategy, and the storage performance corresponding to the data using the replica storage strategy is higher than the storage performance corresponding to the data using the erasure coding storage strategy. The step of adjusting the storage strategy of the target data to an erasure coding storage strategy includes: detecting key parameters of the target data, wherein the key parameters indicate the criticality of the target data in the storage system; determining the number of data blocks and the number of check blocks corresponding to the target data based on the key parameters and the predicted access frequency of the target data; generating a target erasure coding storage strategy for the target data based on the number of data blocks and the number of check blocks, wherein the target data with higher criticality corresponds to more check blocks, and the target data with lower access frequency corresponds to more data blocks and fewer check blocks; the target erasure coding storage strategy is a strategy of dividing the target data into data blocks of the specified number of data blocks and generating check blocks of the specified number of check blocks for storage; and determining the target erasure coding storage strategy as the erasure coding storage strategy.

2. The method according to claim 1, characterized in that, When the storage policy of the target data is adjusted to the replica storage policy, the target change operation on the target data includes: The target data is set to a locked state, wherein the target data in the locked state is only allowed to be modified by one process at any given time. The target data is copied to obtain a temporary data copy, and the temporary data copy is stored in a temporary copy storage node in the storage system, wherein the response speed of the data in the temporary copy storage node to a read request is greater than a preset first speed threshold. The access address of the target data in the storage system is changed to the first access address of the temporary data copy in the temporary copy storage node. At the same time, a target process of the storage system is called to convert the target data from erasure coding storage format to copy storage format to obtain a formal data copy. The formal data copy is then stored in the formal copy storage node of the storage system. The data in the storage system that belongs to the erasure coding storage format is divided into multiple data blocks, and multiple check blocks are generated for multiple data blocks for storage. The data that belongs to the copy storage format is copied into one or more copies for storage. The response speed of the data in the formal copy storage node to the read request is greater than a preset second speed threshold. If the formal data copy has already been stored on the formal copy storage node, the access address of the target data in the storage system is changed to the second access address of the formal data copy in the formal copy storage node, and a deletion operation is performed on the target data stored in the storage system according to the erasure coding storage format, as well as the temporary data copy.

3. The method according to claim 1, characterized in that, When the storage strategy of the target data is adjusted to the erasure coding storage strategy, the target change operation on the target data includes: The target data is converted from a replica storage format to an erasure coding storage format to obtain multiple target data blocks and multiple target verification blocks. In the storage system, the data belonging to the erasure coding storage format is divided into multiple data blocks, and multiple verification blocks are generated for multiple data blocks for storage. The data belonging to the replica storage format is copied into one or more replicas for storage. Multiple target data blocks and multiple target verification blocks are stored in multiple erasure coding storage nodes of the storage system, wherein the storage overhead required to store the data through multiple erasure coding storage nodes is lower than a preset overhead threshold. The access address of the target data in the storage system is changed to a third access address, wherein the third access address is the address required to read multiple target data blocks and multiple target check blocks from multiple erasure coding storage nodes; Perform a deletion operation on the target data stored in the storage system according to the replica storage format.

4. The method according to claim 1, characterized in that, The step of detecting whether the data access type of multiple data items has changed based on the frequency range falling within the access frequency includes: The following steps are used to detect whether the data access type of the reference data has changed, wherein the reference data is any data in the storage system: Obtain the target correspondence between multiple frequency ranges and multiple data access types; Match the reference data access type corresponding to the reference frequency range into which the reference access frequency of the reference data falls from the target correspondence; Detect whether the access type of the reference data is consistent with the current access type of the reference data; If the access type of the reference data is found to be inconsistent with the current access type of the reference data, it is determined that the access type of the reference data has changed.

5. The method according to claim 4, characterized in that, The step of obtaining the target correspondence between the multiple frequency ranges and the multiple data access types includes: The system load parameters of the storage system are detected, wherein the system load parameters are used to indicate the load status of the system resources of the storage system; When the system load parameters fall within a first load range, a first correspondence between multiple frequency ranges and multiple data access types is determined as the target correspondence. The first load range is a load range greater than or equal to a target lower load limit and less than or equal to a target upper load limit. The first correspondence is as follows: a first frequency range corresponds to a hot access type, and a second frequency range corresponds to a cold access type. The multiple frequency ranges include the first frequency range and the second frequency range. The first frequency range is a frequency range with a lower limit equal to a preset target lower frequency limit, and the second frequency range is a frequency range with an upper limit equal to a preset target upper frequency limit. The multiple data access types include the hot access type and the cold access type. The access probability indicated by the hot access type is greater than or equal to a preset probability threshold, and the access probability indicated by the cold access type is less than the preset probability threshold. When the system load parameters fall into the second load range, the lower limit of the target frequency in the first frequency range in the first correspondence is lowered by a first frequency threshold, and the upper limit of the target frequency in the second frequency range is increased by a second frequency threshold to obtain a second correspondence. The second correspondence is determined as the target correspondence, wherein the second load range is a load range that is less than the lower limit of the target load. When the system load parameters fall into the third load range, the lower limit of the target frequency in the first frequency range in the first correspondence is raised to a third frequency threshold, and the upper limit of the target frequency in the second frequency range is lowered to a fourth frequency threshold to obtain the third correspondence. The third correspondence is determined as the target correspondence, wherein the third load range is a load range greater than the upper limit of the target load.

6. The method according to claim 1, characterized in that, The method of predicting the access frequency of multiple data items within a future time period based on their historical access characteristics in the storage system includes: The access frequency of the i-th data among N data in the storage system will be predicted within a future time period using the following steps, where N is an integer greater than or equal to 1, and i is a positive integer less than or equal to N: Multiple rounds of sampling are performed on the access information of the i-th data within the historical time period to obtain the i-th access feature sequence corresponding to the i-th data, wherein the access information includes records of the i-th data being accessed within the historical time period; The i-th access feature sequence is input into the target prediction model to obtain the i-th access frequency output by the target prediction model, wherein the target prediction model is obtained by training the initial prediction model using access feature sequence samples labeled with access frequency tags; The i-th access frequency is determined as the access frequency of the i-th data.

7. The method according to claim 6, characterized in that, The step of sampling the access information of the i-th data within the historical time period in multiple rounds to obtain the i-th access feature sequence corresponding to the i-th data includes: Obtain the preset sampling time interval; The features of the access information of the i-th data within the historical time period are sampled according to the sampling time interval to obtain multiple historical access features; The historical access features are sorted according to the sampling time order to obtain the i-th access feature sequence.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data storage control method as described in any one of claims 1 to 7.

Images