Data processing method and device of storage cache system, equipment and medium

Abstract

The embodiment of the invention provides a data processing method and device for a storage cache system, equipment and a medium, and the method comprises the steps: when a write operation instruction for a target object is received, creating a snapshot object for a cache object corresponding to the target object in a cache according to data and metadata carried by the write operation instruction, updating a cache object corresponding to the target object in the cache; persistently storing the cache object and the snapshot object corresponding to the target object; when a read operation instruction for the target object is received, the data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the persistently stored cache object and the persistently stored snapshot object, so that the access speed in a high-concurrency environment is improved, and the access pressure on a rear-end storage system is relieved.

Description

Technical Field

[0001] The present invention belongs to the field of storage technology, and specifically relates to a data processing method for a storage cache system, a data processing device for a storage cache system, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the era of cloud computing, disaster recovery and backup of massive data are becoming increasingly important. Snapshots have become an important means of disaster recovery and backup due to their advantages such as fast execution time and small disk space usage.

[0003] In enterprise-level storage systems, snapshots are an essential feature. Snapshots are generally implemented using copy-on-write technology (COW), which is a read-only image of a data object at a certain moment. Before modifying or writing the data object, a copy of the data object is made, and then the data object is modified and written. The copied data object is the snapshot. However, existing snapshot services usually face problems such as large-scale data replication and storage, write amplification, and the number of accesses to slow storage media such as hard disks, resulting in slow access speeds in high-concurrency environments, excessive pressure on the backend storage system, and memory usage.

[0004] Caching technology is a technology that reduces the number of accesses to the original data source by saving data copies. It reduces the number of accesses to slow storage media by storing frequently accessed data in high-speed storage media, thereby increasing data access speed. Summary of the invention

[0005] In view of the relevant problems in the above-mentioned background technology, embodiments of the present invention are proposed to provide a data processing method for a storage cache system, a data processing device for a storage cache system, an electronic device and a computer-readable storage medium that overcome the above-mentioned problems or at least partially solve the above-mentioned problems.

[0006] In order to solve the above problem, an embodiment of the present invention discloses a data processing method of a storage cache system, which is applied to the storage cache system. The method includes:

[0007] When a write operation instruction for a target object is received, a snapshot object is created for a cache object corresponding to the target object in the cache according to the data and metadata carried by the write operation instruction, and the cache object corresponding to the target object in the cache is updated;

[0008] Persistently storing the cache object and snapshot object corresponding to the target object;

[0009] When a read operation instruction for the target object is received, data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage.

[0010] Optionally, creating a snapshot object for the cache object corresponding to the target object in the cache includes:

[0011] Obtain data and metadata of a cache object corresponding to the target object in the cache;

[0012] Comparing the snapshot version in the data carried by the write operation instruction with the snapshot version in the data of the cache object corresponding to the target object in the cache;

[0013] When the snapshot version in the data carried by the write operation instruction and the snapshot version in the data of the cache object corresponding to the target object in the cache are different, creating a snapshot object for the cache object corresponding to the target object;

[0014] The data and metadata in the cache object corresponding to the target object are written into the snapshot object, and the snapshot version in the data carried by the write operation instruction is written into the snapshot object.

[0015] Optionally, updating the cache object corresponding to the target object in the cache includes:

[0016] After the snapshot object is created, the data in the cache object is cleared, and the data and metadata carried in the write operation instruction are written into the cache object.

[0017] Optionally, the persistently storing the cache object and the snapshot object corresponding to the target object includes:

[0018] Determine the groups to which the cache object and the snapshot object belong;

[0019] Aggregate the cache object and the snapshot object according to the group to which they belong;

[0020] The aggregated data obtained by the aggregation processing is put into a preset group persistence queue, and the aggregated data in the preset group persistence queue is persisted.

[0021] Optionally, persisting the aggregated data in the preset group persistence queue includes:

[0022] When a preset persistence condition is met, the aggregated data in the preset group persistence queue is divided into multiple copies as persistence subtasks according to a preset data threshold; the persistence condition is that the size of the preset group persistence queue exceeds a preset size threshold or the time from the last persistence operation exceeds a preset time threshold;

[0023] The persistence subtask is subjected to concurrent persistence processing.

[0024] Optionally, when receiving a read operation instruction for the target object, reading data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage includes:

[0025] When the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction exists in the cache, determining a hit condition of the read operation instruction; the hit condition includes that the demand data indicated by the read operation instruction is fully hit in the cache object and the snapshot object of the cache, the demand data indicated by the read operation instruction is partially hit in the cache object and the snapshot object of the cache, or the demand data indicated by the read operation instruction is fully missed in the cache object and the snapshot object of the cache;

[0026] If the hit condition is that the required data indicated by the read operation instruction hits all the cache objects and the snapshot objects of the cache, then reading the required data indicated by the read operation instruction from the cache objects and the snapshot objects of the cache;

[0027] If the hit condition is that the demand data indicated by the read operation instruction partially hits the cache object and the snapshot object of the cache, then read part of the demand data indicated by the read operation instruction from the cache object and the snapshot object of the cache; and read part of the demand data indicated by the read operation instruction from the cache object of persistent storage and the snapshot object of persistent storage;

[0028] If the hit condition is that the required data indicated by the read operation instruction is not hit in the cache object and the snapshot object of the cache, the required data indicated by the read operation instruction is read from the cache object of persistent storage and the snapshot object of persistent storage.

[0029] Optionally, when receiving a read operation instruction for the target object, reading data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage, further includes:

[0030] When the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction does not exist in the cache, but a corresponding snapshot object exists, determining the latest cache object corresponding to the demand data in persistent storage;

[0031] Comparing the snapshot version of the demand data indicated by the read operation instruction with the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0032] When the snapshot version of the demand data indicated by the read operation instruction is greater than the snapshot version of the latest cache object corresponding to the demand data in the persistent storage, the demand data indicated by the read operation instruction is read from the latest cache object corresponding to the demand data in the persistent storage;

[0033] When the snapshot version of the demand data indicated by the read operation instruction is less than or equal to the snapshot version of the latest cache object corresponding to the demand data in the persistent storage, obtaining the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage;

[0034] The demand data indicated by the read operation instruction is read from the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage.

[0035] The embodiment of the present invention further discloses a data processing device for a storage cache system, which is applied to the storage cache system. The device includes:

[0036] A write operation module, configured to, when receiving a write operation instruction for a target object, create a snapshot object for a cache object corresponding to the target object in the cache according to data and metadata carried by the write operation instruction, and update the cache object corresponding to the target object in the cache;

[0037] A persistence module, used for persistently storing the cache object and snapshot object corresponding to the target object;

[0038] A read operation module is used to read the data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage when receiving a read operation instruction for the target object.

[0039] Optionally, the write operation module includes:

[0040] A data acquisition submodule, used to acquire data and metadata of a cache object corresponding to the target object in the cache;

[0041] A first comparison submodule, configured to compare a snapshot version in the data carried by the write operation instruction with a snapshot version in the data of a cache object corresponding to the target object in the cache;

[0042] an object creation submodule, configured to create a snapshot object for the cache object corresponding to the target object when the snapshot version in the data carried by the write operation instruction and the snapshot version in the data of the cache object corresponding to the target object in the cache are different;

[0043] The first writing submodule is used to write the data and metadata in the cache object corresponding to the target object into the snapshot object, and write the snapshot version in the data carried by the write operation instruction into the snapshot object.

[0044] Optionally, the write operation module includes:

[0045] The second writing submodule is used to clear the data in the cache object after creating the snapshot object, and write the data and metadata carried in the write operation instruction into the cache object.

[0046] Optionally, the persistence module includes:

[0047] A group determination submodule, used to determine the groups to which the cache object and the snapshot object belong;

[0048] An aggregation processing submodule, used for performing aggregation processing on the cache object and the snapshot object according to the group to which they belong;

[0049] The persistence submodule is used to put the aggregated data obtained by the aggregation processing into the preset group persistence queue, and persist the aggregated data in the preset group persistence queue.

[0050] Optionally, the persistence submodule includes:

[0051] A task division unit, used for dividing the aggregated data in the preset group persistence queue into multiple copies as persistence subtasks according to a preset data threshold when a preset persistence condition is met; the persistence condition is that the size of the preset group persistence queue exceeds a preset size threshold or the time from the last persistence operation exceeds a preset time threshold;

[0052] The persistence unit is used to perform concurrent persistence processing on the persistence subtasks.

[0053] Optionally, the read operation module includes:

[0054] a hit situation determination submodule, used for determining the hit situation of the read operation instruction when the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction exists in the cache; the hit situation includes that the demand data indicated by the read operation instruction is fully hit in the cache object and the snapshot object of the cache, the demand data indicated by the read operation instruction is partially hit in the cache object and the snapshot object of the cache, or the demand data indicated by the read operation instruction is completely missed in the cache object and the snapshot object of the cache;

[0055] A first reading submodule, configured to read the demand data indicated by the read operation instruction from the cache object and the snapshot object of the cache if the hit condition is that the demand data indicated by the read operation instruction hits all of the cache object and the snapshot object of the cache;

[0056] A second reading submodule is configured to read part of the required data indicated by the read operation instruction from the cache object and the snapshot object of the cache if the hit condition is that the required data indicated by the read operation instruction partially hits the cache object and the snapshot object of the cache; and read part of the required data indicated by the read operation instruction from the cache object of persistent storage and the snapshot object of persistent storage;

[0057] The third reading submodule is used to read the required data indicated by the read operation instruction from the cache object stored persistently and the snapshot object stored persistently if the hit condition is that the required data indicated by the read operation instruction is not hit in the cache object and the snapshot object of the cache.

[0058] Optionally, the read operation module further includes:

[0059] an object determination submodule, configured to determine the latest cache object corresponding to the demand data in persistent storage when the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction does not exist in the cache, but a corresponding snapshot object exists;

[0060] A second version comparison module, configured to compare the snapshot version of the demand data indicated by the read operation instruction with the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0061] a fourth reading submodule, configured to read the demand data indicated by the read operation instruction from the latest cache object corresponding to the demand data in the persistent storage when the snapshot version of the demand data indicated by the read operation instruction is greater than the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0062] An object acquisition submodule, configured to acquire a snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage when the snapshot version of the demand data indicated by the read operation instruction is less than or equal to the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0063] The fifth reading submodule is used to read the demand data indicated by the read operation instruction from the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage.

[0064] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the data processing method embodiment of the above-mentioned storage cache system when executed by the processor.

[0065] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the data processing method embodiment of the above-mentioned storage cache system are implemented.

[0066] The embodiments of the present invention include the following advantages:

[0067] In the embodiment of the present invention, when the storage cache system receives a write operation instruction for the target object, the storage cache system creates a snapshot object for the cache object corresponding to the target object in the cache according to the data and metadata carried by the write operation instruction, and updates the cache object corresponding to the target object in the cache; it can ensure that the state of the target object is completely recorded before the cache object is updated. In this way, even if an error or system failure occurs during the update process, the state before the update can be restored through the snapshot object, thereby maintaining data consistency. The existence of the snapshot object provides a fast channel for data recovery. When necessary, data can be directly restored from the snapshot, which greatly shortens the recovery time; when receiving a read operation instruction for the target object, the data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage, thereby improving the access speed in a high-concurrency environment and reducing the access pressure on the back-end storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flowchart of a data processing method for a storage cache system according to an embodiment of the present invention;

[0069] Figure 2is a structural block diagram of a data processing device for a storage cache system provided by an embodiment of the present invention;

[0070] Figure 3 is a structural block diagram of an electronic device provided by an embodiment of the present invention;

[0071] Figure 4 It is a structural block diagram of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] Among storage access requests, write and read are two basic and important operations, which involve saving data into a storage device and retrieving data from a storage device, respectively.

[0074] Caching technology plays an important role in data storage and access. It is a key technology to improve system performance and reduce the number of accesses to external resources. It stores some frequently used data in memory so that it can be accessed quickly when needed. By caching objects, the system can reduce the number of accesses to slow storage devices such as disks and databases, thereby improving response speed and throughput.

[0075] Snapshot technology is a data copy technology based on a time point. Its core is to be able to record and save data information at a certain moment so as to restore the data to the state at that point in time when needed.

[0076] Metadata is data that describes data, also known as intermediary data or relay data. It is mainly used to describe the attribute information of data to support functions such as indicating storage location, historical data, resource search, file records, etc. Metadata plays an important role in data management and applications.

[0077] One of the core concepts of the embodiments of the present invention is to accelerate the access response to storage access requests by combining the cache technology and the snapshot technology.

[0078] Reference Figure 1 , shows a flowchart of a data processing method for a storage cache system according to an embodiment of the present invention, the method is applied to the storage cache system, and the method may specifically include the following steps:

[0079] Step 101, when a write operation instruction for a target object is received, a snapshot object is created for a cache object corresponding to the target object in a cache according to data and metadata carried by the write operation instruction, and the cache object corresponding to the target object in the cache is updated;

[0080] In a storage cache system, when a write operation instruction is received for a target object, it is very important to take a series of measures to ensure data consistency and recoverability. One effective strategy is to create a snapshot object at the cache level and keep the state of the old data while updating the cache object.

[0081] In one embodiment, the step 101 of creating a snapshot object for the cache object corresponding to the target object in the cache may include the following sub-steps:

[0082] Sub-step S11, obtaining data and metadata of a cache object corresponding to the target object in the cache;

[0083] The system searches the cache for the cache object corresponding to the target object and obtains the data and metadata of the cache object.

[0084] Sub-step S12, comparing the snapshot version in the data carried by the write operation instruction with the snapshot version in the data of the cache object corresponding to the target object in the cache;

[0085] The snapshot version is used to indicate the data version operated at the current time;

[0086] Sub-step S13, when the snapshot version in the data carried by the write operation instruction and the snapshot version in the data of the cache object corresponding to the target object in the cache are different, creating a snapshot object for the cache object corresponding to the target object;

[0087] When the snapshot versions are different, the system creates a snapshot object based on the current state of the cached object, including data and metadata. This snapshot object is a complete copy of the current state of the target object, which contains all the information of the target object before the write operation occurs.

[0088] Sub-step S14, writing the data and metadata in the cache object corresponding to the target object into the snapshot object, and writing the snapshot version in the data carried by the write operation instruction into the snapshot object.

[0089] The created snapshot object will additionally store the snapshot version carried by the write operation to facilitate the rollback of system data;

[0090] In the above embodiment, the correctness of the data update can be ensured by comparing the snapshot version in the data carried by the write operation instruction with the snapshot version of the existing cache object in the cache. If the versions do not match, the system can identify potential data conflicts or inconsistencies and take corresponding measures such as creating a new snapshot object to protect the integrity and consistency of the data. At the same time, the snapshot object is a complete copy of the state of the cache object at a certain point in time. The embodiment of the present invention allows the system to roll back to a previous state at any time by additionally storing the snapshot version before the update, which is very useful for scenarios such as fault recovery, data auditing or testing.

[0091] In one embodiment, the updating of the cache object corresponding to the target object in the cache in step 101 may include the following sub-steps:

[0092] Sub-step S21, after creating the snapshot object, clearing the data in the cache object, and writing the data and metadata carried in the write operation instruction into the cache object.

[0093] After the snapshot object is created, the system uses the data and metadata carried in the write operation instruction to update the cache object; this update process may include operations such as modifying the object's attributes, adding new attributes, or deleting old attributes.

[0094] Step 102, persistently storing the cache object and snapshot object corresponding to the target object;

[0095] Persistent storage of cache objects and snapshot objects corresponding to the target object is an important step to ensure data reliability and recoverability. Persistent storage means transferring data from volatile storage media such as cache to non-volatile storage media such as persistence layer so that data can be recovered in case of system restart, failure or data loss.

[0096] In one embodiment, step 102 may include the following sub-steps:

[0097] Sub-step S31, determining the groups to which the cache object and the snapshot object belong;

[0098] In a storage cache system, the management of cache objects and snapshot objects usually involves organizing them into logical groups or collections, namespaces, etc. for easy management, access, and persistent storage.

[0099] Sub-step S32, performing aggregation processing on the cache object and the snapshot object according to the group to which they belong;

[0100] Aggregating cache objects and snapshot objects according to their groups helps optimize storage structure, improve data access efficiency, and simplify management operations. The aggregation strategy can be physical aggregation or logical aggregation.

[0101] Sub-step S33, putting the aggregated data obtained by the aggregation processing into a preset group persistence queue, and persisting the aggregated data in the preset group persistence queue.

[0102] In the above embodiment, by aggregating cache objects and snapshot objects according to their groups, and putting the aggregated data into a preset group persistence queue for persistence, data redundancy can be reduced, thereby improving the efficiency of data management. Aggregation based on the group also helps to achieve fast positioning during access, and in the aggregation process, the version information of each cache object and snapshot object can be conveniently recorded. This helps to maintain data consistency and integrity when data is updated, and prevent data conflicts and losses.

[0103] In one embodiment, the sub-step S33 may further include the following sub-steps:

[0104] Sub-step S331, when a preset persistence condition is met, the aggregated data in the preset group persistence queue is divided into multiple copies as persistence subtasks according to a preset data threshold; the persistence condition is that the size of the preset group persistence queue exceeds a preset size threshold or the time from the last persistence operation exceeds a preset time threshold;

[0105] The preset data threshold can be adjusted dynamically according to business needs, and the system can monitor the status of the group persistence queue and automatically divide the subtasks and perform persistence operations when the persistence conditions are met. This simplifies management operations and reduces the need for manual intervention.

[0106] Sub-step S332, performing concurrent persistence processing on the persistence subtask.

[0107] Through concurrent processing, the system can handle multiple persistence subtasks at the same time, which significantly reduces the total time of data persistence and better utilizes system resources.

[0108] Step 103, when a read operation instruction for the target object is received, data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage.

[0109] In a storage cache system, when a read operation instruction for a target object is received, the system needs to efficiently locate and read the data of the target object. Depending on the situation where the data may exist in both the cache and the persistent storage, the system's read strategy is also different.

[0110] In one embodiment, step 103 may include the following sub-steps:

[0111] Sub-step S41, when the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction exists in the cache, determining a hit condition of the read operation instruction; the hit condition includes that the demand data indicated by the read operation instruction is fully hit in the cache object and the snapshot object of the cache, the demand data indicated by the read operation instruction is partially hit in the cache object and the snapshot object of the cache, or the demand data indicated by the read operation instruction is fully missed in the cache object and the snapshot object of the cache;

[0112] When the required data indicated by the read operation instruction is completely available in the cache object of the cache, it can be called a full hit;

[0113] When part of the required data indicated by the read operation instruction is available in the cache object of the cache, but part of the data is older or does not exist in the current cache, but exists in the snapshot object, it can be called a partial hit;

[0114] When the required data indicated by the read operation instruction is not available in both the cache object and the snapshot object of the cache, it can be called a total miss.

[0115] Sub-step S42, if the hit condition is that the required data indicated by the read operation instruction hits all the cache objects and the snapshot objects of the cache, then reading the required data indicated by the read operation instruction from the cache objects and the snapshot objects of the cache;

[0116] When all hits occur, it means that all requested data is up-to-date and can be returned directly from the cache without further querying the persistent store.

[0117] Sub-step S43, if the hit condition is that the demand data indicated by the read operation instruction partially hits the cache object and the snapshot object of the cache, then read part of the demand data indicated by the read operation instruction from the cache object and the snapshot object of the cache; and read part of the demand data indicated by the read operation instruction from the cache object of persistent storage and the snapshot object of persistent storage;

[0118] When there is a partial hit, the reading process is as follows:

[0119] Read the available partial data from the cache object of the cache, and read the remaining available partial data from the cache snapshot object. For the missing data in the cache and snapshot objects, the system needs to query the persistent storage; for the missing data part, the system will then query the metadata cache; the metadata cache usually contains indexes or entries about data location, version, size and other information, parse the metadata cache, use the information parsed from the metadata cache, read the data on the persistent storage through the persistence layer, and finally return the results of the cache query together with the data read in the persistent storage as the required data.

[0120] Sub-step S44, if the hit condition is that the required data indicated by the read operation instruction is not hit in the cache object and the snapshot object of the cache, then read the required data indicated by the read operation instruction from the persistently stored cache object and the persistently stored snapshot object.

[0121] When all misses occur, the reading process is as follows: all required data is read from the persistent storage through the metadata cache.

[0122] In the above embodiment, when the required data completely hits the cache, the system can read the data directly from the cache, avoiding the overhead of accessing the persistent storage, thereby improving the efficiency of data access; when the required data partially hits the cache, the system can intelligently read partial data from the cache and the persistent storage respectively, and then merge the data to meet the requirements of the read operation instruction. This flexibility ensures that the system can provide the required data even when the cache does not fully match.

[0123] In one embodiment, step 103 may further include the following sub-steps:

[0124] Sub-step S45, when the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction does not exist in the cache, but a corresponding snapshot object exists, determining the latest cache object corresponding to the demand data in persistent storage;

[0125] This happens when the cache object has been flushed but the snapshot object has not yet been flushed.

[0126] Sub-step S46, comparing the snapshot version of the demand data indicated by the read operation instruction with the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0127] Determine the read strategy that should be selected in this case by comparing snapshot versions;

[0128] Sub-step S47, when the snapshot version of the demand data indicated by the read operation instruction is greater than the snapshot version of the latest cache object corresponding to the demand data in the persistent storage, the demand data indicated by the read operation instruction is read from the latest cache object corresponding to the demand data in the persistent storage;

[0129] Sub-step S48, when the snapshot version of the demand data indicated by the read operation instruction is less than or equal to the snapshot version of the latest cache object corresponding to the demand data in the persistent storage, obtaining the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage;

[0130] When there is no cache object corresponding to the required data version in the cache, the system can correctly determine the latest cache object in the persistent storage and decide where to read the data by comparing the snapshot version. This mechanism ensures that even if the data in the cache is outdated or does not exist, the system can obtain consistent and accurate data from the persistent storage; by comparing the snapshot version of the required data indicated by the read operation instruction with the snapshot version of the cache object in the persistent storage, the system can support access to historical version data. This is very useful for scenarios that require data backtracking or data analysis.

[0131] Sub-step S49, reading the demand data indicated by the read operation instruction from the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage.

[0132] In the above embodiment, when the storage cache system receives a write operation instruction for the target object, the storage cache system creates a snapshot object for the cache object corresponding to the target object in the cache according to the data and metadata carried by the write operation instruction, and updates the cache object corresponding to the target object in the cache; it can ensure that the state of the target object is completely recorded before the cache object is updated. In this way, even if an error or system failure occurs during the update process, the snapshot object can be used to restore to the state before the update, thereby maintaining data consistency. The existence of the snapshot object provides a fast channel for data recovery. When necessary, data can be directly restored from the snapshot, greatly shortening the recovery time; when receiving a read operation instruction for the target object, the data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage. This flexible data access method can be selected according to different application scenarios and requirements to meet data consistency requirements in different scenarios.

[0133] In an embodiment of the present invention, the method may further include:

[0134] After persistently storing the cache object and the snapshot object corresponding to the target object, obtaining response return information, the response return information including the data and metadata of the cache object for the persistence operation and the data and metadata of the snapshot object group;

[0135] Determine a corresponding metadata cache in the cache according to the persisted data and metadata of the cache object;

[0136] Obtaining a version identifier of the metadata of the cache object after the persistence and a version identifier of the metadata of the corresponding metadata cache in the cache;

[0137] If the version identifier of the metadata of the cache object after persistence is different from the version identifier of the metadata of the corresponding metadata cache in the cache, the corresponding metadata cache in the cache is updated; by updating the metadata cache, the reading pressure in the storage cache system is reduced.

[0138] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0139] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] Reference Figure 2 , shows a structural block diagram of a data processing device of a storage cache system provided by an embodiment of the present invention, the device is applied to the storage cache system, and the device includes:

[0141] The write operation module 201 is used to create a snapshot object for the cache object corresponding to the target object in the cache according to the data and metadata carried by the write operation instruction when receiving the write operation instruction for the target object, and update the cache object corresponding to the target object in the cache;

[0142] The persistence module 202 is used to persistently store the cache object and the snapshot object corresponding to the target object;

[0143] The read operation module 203 is used to read the data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage when receiving a read operation instruction for the target object.

[0144] In one embodiment, the write operation module includes:

[0145] A data acquisition submodule, used to acquire data and metadata of a cache object corresponding to the target object in the cache;

[0146] A first comparison submodule, configured to compare a snapshot version in the data carried by the write operation instruction with a snapshot version in the data of a cache object corresponding to the target object in the cache;

[0147] an object creation submodule, configured to create a snapshot object for the cache object corresponding to the target object when the snapshot version in the data carried by the write operation instruction and the snapshot version in the data of the cache object corresponding to the target object in the cache are different;

[0148] The first writing submodule is used to write the data and metadata in the cache object corresponding to the target object into the snapshot object, and write the snapshot version in the data carried by the write operation instruction into the snapshot object.

[0149] In one embodiment, the write operation module includes:

[0150] The second writing submodule is used to clear the data in the cache object after creating the snapshot object, and write the data and metadata carried in the write operation instruction into the cache object.

[0151] In one embodiment, the persistence module includes:

[0152] A group determination submodule, used to determine the groups to which the cache object and the snapshot object belong;

[0153] An aggregation processing submodule, used for performing aggregation processing on the cache object and the snapshot object according to the group to which they belong;

[0154] The persistence submodule is used to put the aggregated data obtained by the aggregation processing into the preset group persistence queue, and persist the aggregated data in the preset group persistence queue.

[0155] In one embodiment, the persistence submodule includes:

[0156] A task division unit, used for dividing the aggregated data in the preset group persistence queue into multiple copies as persistence subtasks according to a preset data threshold when a preset persistence condition is met; the persistence condition is that the size of the preset group persistence queue exceeds a preset size threshold or the time from the last persistence operation exceeds a preset time threshold;

[0157] The persistence unit is used to perform concurrent persistence processing on the persistence subtasks.

[0158] In one embodiment, the read operation module includes:

[0159] a hit situation determination submodule, used for determining the hit situation of the read operation instruction when the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction exists in the cache; the hit situation includes that the demand data indicated by the read operation instruction is fully hit in the cache object and the snapshot object of the cache, the demand data indicated by the read operation instruction is partially hit in the cache object and the snapshot object of the cache, or the demand data indicated by the read operation instruction is completely missed in the cache object and the snapshot object of the cache;

[0160] A first reading submodule, configured to read the demand data indicated by the read operation instruction from the cache object and the snapshot object of the cache if the hit condition is that the demand data indicated by the read operation instruction hits all of the cache object and the snapshot object of the cache;

[0161] A second reading submodule is configured to read part of the required data indicated by the read operation instruction from the cache object and the snapshot object of the cache if the hit condition is that the required data indicated by the read operation instruction partially hits the cache object and the snapshot object of the cache; and read part of the required data indicated by the read operation instruction from the cache object of persistent storage and the snapshot object of persistent storage;

[0162] The third reading submodule is used to read the required data indicated by the read operation instruction from the cache object stored persistently and the snapshot object stored persistently if the hit condition is that the required data indicated by the read operation instruction is not hit in the cache object and the snapshot object of the cache.

[0163] In one embodiment, the read operation module further includes:

[0164] an object determination submodule, configured to determine the latest cache object corresponding to the demand data in persistent storage when the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction does not exist in the cache, but a corresponding snapshot object exists;

[0165] A second version comparison module, configured to compare the snapshot version of the demand data indicated by the read operation instruction with the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0166] a fourth reading submodule, configured to read the demand data indicated by the read operation instruction from the latest cache object corresponding to the demand data in the persistent storage when the snapshot version of the demand data indicated by the read operation instruction is greater than the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0167] An object acquisition submodule, configured to acquire a snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage when the snapshot version of the demand data indicated by the read operation instruction is less than or equal to the snapshot version of the latest cache object corresponding to the demand data in the persistent storage;

[0168] The fifth reading submodule is used to read the demand data indicated by the read operation instruction from the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage.

[0169] In one embodiment, the device further comprises:

[0170] A response return information acquisition module, used to acquire response return information after persistently storing the cache object and snapshot object corresponding to the target object, wherein the response return information includes data and metadata of the cache object for which the persistence operation is performed and data and metadata of the snapshot object group;

[0171] A cache determination module, used to determine the corresponding metadata cache in the cache according to the persisted data and metadata of the cache object;

[0172] A version identification acquisition module, used to acquire the version identification of the metadata of the cache object after the persistence, and the version identification of the metadata of the corresponding metadata cache in the cache;

[0173] The cache update module is used to update the corresponding metadata cache in the cache if the version identifier of the metadata of the cache object after persistence is different from the version identifier of the metadata of the corresponding metadata cache in the cache.

[0174] In the above embodiment, when the storage cache system receives a write operation instruction for the target object, the storage cache system creates a snapshot object for the cache object corresponding to the target object in the cache according to the data and metadata carried by the write operation instruction, and updates the cache object corresponding to the target object in the cache; it can ensure that the state of the target object is completely recorded before the cache object is updated. In this way, even if an error or system failure occurs during the update process, the snapshot object can be used to restore to the state before the update, thereby maintaining data consistency. The existence of the snapshot object provides a fast channel for data recovery. When necessary, data can be directly restored from the snapshot, greatly shortening the recovery time; when receiving a read operation instruction for the target object, the data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage. Thereby improving the access speed in a high-concurrency environment and reducing the access pressure on the back-end storage system.

[0175] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] like Figure 3 , showing a structural block diagram of an electronic device 30 provided in an embodiment of the present invention, including:

[0177] A processor 301, a memory 302, and a computer program 3021 stored in the memory 302 and capable of running on the processor 301. When the computer program 3021 is executed by the processor 301, each process of the data processing method embodiment of the above-mentioned storage cache system is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0178] like Figure 4 , showing a structural block diagram of a computer-readable storage medium 40 provided in an embodiment of the present invention, a computer program 401 is stored on the computer-readable storage medium 40, and when the computer program 401 is executed by the processor, each process of the data processing method embodiment of the above-mentioned storage cache system is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0179] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0180] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0181] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0184] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0185] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0186] The data processing method of a storage cache system, a data processing device of a storage cache system, an electronic device and a computer-readable storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method for a storage cache system, applied to a storage cache system, characterized in that: The method comprises: When a write operation instruction for a target object is received, a snapshot object is created for a cache object corresponding to the target object in the cache according to the data and metadata carried by the write operation instruction, and the cache object corresponding to the target object in the cache is updated; Persistently storing the cache object and snapshot object corresponding to the target object; When a read operation instruction for the target object is received, data of the target object is read from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage.

2. The method according to claim 1, characterized in that The step of creating a snapshot object for the cache object corresponding to the target object in the cache includes: Obtain data and metadata of a cache object corresponding to the target object in the cache; Compare the snapshot version in the data carried by the write operation instruction with the target in the cache The snapshot version of the data in the cache object corresponding to the object; When the snapshot version of the data carried by the write operation instruction and the target object in the cache When the snapshot versions in the data of the corresponding cache objects are different, creating a snapshot object for the cache object corresponding to the target object; The data and metadata in the cache object corresponding to the target object are written into the snapshot object, and the snapshot version in the data carried by the write operation instruction is written into the snapshot object.

3. The method according to claim 2, characterized in that The updating of the cache object corresponding to the target object in the cache includes: After creating the snapshot object, clear the data in the cache object and write the The data and metadata carried in the instruction are written into the cache object.

4. The method according to claim 1, characterized in that: The persistent storage of the cache object and the snapshot object corresponding to the target object includes: Determine the groups to which the cache object and the snapshot object belong; Aggregate the cache object and the snapshot object according to the group to which they belong; The aggregated data obtained by the aggregation processing is put into a preset group persistence queue, and the aggregated data in the preset group persistence queue is persisted.

5. The method according to claim 4, characterized in that The persisting the aggregated data in the preset group persistence queue includes: When a preset persistence condition is met, the aggregated data in the preset group persistence queue is divided into multiple copies as persistence subtasks according to a preset data threshold; the persistence condition is that the size of the preset group persistence queue exceeds a preset size threshold or the time from the last persistence operation exceeds a preset time threshold; Concurrent persistence processing is performed on the persistence subtask.

6. The method according to claim 1, characterized in that When receiving a read operation instruction for the target object, reading data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage, comprises: When the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction exists in the cache, determining a hit condition of the read operation instruction; the hit condition includes that the demand data indicated by the read operation instruction is fully hit in the cache object and the snapshot object of the cache, the demand data indicated by the read operation instruction is partially hit in the cache object and the snapshot object of the cache, or the demand data indicated by the read operation instruction is fully missed in the cache object and the snapshot object of the cache; If the hit condition is that the required data indicated by the read operation instruction hits all the cache objects and the snapshot objects of the cache, then reading the required data indicated by the read operation instruction from the cache objects and the snapshot objects of the cache; If the hit condition is that the demand data indicated by the read operation instruction partially hits the cache object and the snapshot object of the cache, then read part of the demand data indicated by the read operation instruction from the cache object and the snapshot object of the cache; and read part of the demand data indicated by the read operation instruction from the cache object of persistent storage and the snapshot object of persistent storage; If the hit condition is that the required data indicated by the read operation instruction is not hit in the cache object and the snapshot object of the cache, the required data indicated by the read operation instruction is read from the cache object of persistent storage and the snapshot object of persistent storage.

7. The method according to claim 6, characterized in that When receiving a read operation instruction for the target object, reading data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object in persistent storage, and the snapshot object in persistent storage, further comprising: When the cache object corresponding to the snapshot version of the demand data indicated by the read operation instruction does not exist in the cache, but a corresponding snapshot object exists, determining the latest cache object corresponding to the demand data in persistent storage; Comparing the snapshot version of the demand data indicated by the read operation instruction with the snapshot version of the latest cache object corresponding to the demand data in the persistent storage; When the snapshot version of the demand data indicated by the read operation instruction is greater than the snapshot version of the latest cache object corresponding to the demand data in the persistent storage, the demand data indicated by the read operation instruction is read from the latest cache object corresponding to the demand data in the persistent storage; When the snapshot version of the demand data indicated by the read operation instruction is less than or equal to the snapshot version of the latest cache object corresponding to the demand data in the persistent storage, obtaining the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage; The demand data indicated by the read operation instruction is read from the snapshot object corresponding to the latest cache object corresponding to the demand data in the persistent storage.

8. A data processing device for a storage cache system, applied to the storage cache system, characterized in that: The device comprises: A write operation module, configured to, when receiving a write operation instruction for a target object, create a snapshot object for a cache object corresponding to the target object in the cache according to data and metadata carried by the write operation instruction, and update the cache object corresponding to the target object in the cache; A persistence module, used to persistently store the cache object and snapshot object corresponding to the target object; a read operation module, used to read the data of the target object from at least one of the cache object in the cache, the snapshot object in the cache, the cache object stored persistently, and the snapshot object stored persistently when receiving a read operation instruction for the target object.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method of the storage cache system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method of the storage cache system according to any one of claims 1 to 7 are implemented.

Images