Data processing method and device based on multi-layer cache, and electronic device

By using unique identification code to manage the object mapping relationship between the cache layer and the data layer in the distributed hierarchical cache architecture, the problem of transactional processing complexity during snapshot rename and deletion is solved, and the efficiency and performance of data processing are improved.

CN114780043BActive Publication Date: 2024-12-06BEIJING XSKY TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210497433.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-09
Publication Date
2024-12-06
Estimated Expiration
2042-05-09

AI Technical Summary

Technical Problem

In a distributed hierarchical cache architecture, the transactional processing of snapshot rename and deletion processes is complex and has high latency, making it difficult to meet performance requirements, and the prior art has failed to effectively solve this problem.

Method used

By introducing unique identification codes into the placement group PG, the object mapping relationship between the cache layer and the data layer is established, the migration and management of cache objects and data objects are realized, the complexity of transactional processing is reduced, and the mapping relationship is adjusted when needed is adjusted. To support snapshot rename and delete operations.

Benefits of technology

It improves data query efficiency and cache efficiency, significantly reduces transactional processing delay, solves the problem of transactional processing complexity in distributed hierarchical cache architecture, and improves the performance and consistency of data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114780043B_ABST
    Figure CN114780043B_ABST
Patent Text Reader

Abstract

The present invention discloses a data processing method and device, and an electronic device based on multi-layer cache. The method includes: receiving a data write request sent by a client, wherein the data write request carries at least: target data to be written; responding to the data write request, generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata; migrating the cache object into the cache layer, and establishing a first mapping relationship between the object metadata of the cache object and the unique identification code; storing the data object in the data layer, and establishing a second mapping relationship between the object metadata of the data object and the unique identification code. The present invention solves the technical problem of complex transactional processing methods when using a layered cache architecture for data caching in related technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and device based on multi-layer cache, and an electronic device. Background Art

[0002] In the related technologies, snapshots have become an indispensable and common solution in storage or application software disaster recovery solutions, and are the basis for replication and backup of other systems, and have become a standard feature of storage. There are two common snapshot implementation principles: COW (Copy On Write) and ROW (Redirect On Write). Among them, ROW is implemented by renaming the snapshot, creating new metadata and establishing a new mapping relationship each time a snapshot is generated. It is relatively easy to implement in the local cache. The metadata information of all objects is recorded in the cache. Usually the cache and the data disk are in the same node (that is, a centralized cache, the cache and the data disk have the same redundancy and are bound to each other). When the snapshot is renamed, when the object metadata information in the high-speed medium is updated, the generated snapshot needs to point to the source data object, and a series of data updates will be performed. Because there is only a single-layer structure and only one primary PG, through strong consistency guarantee, other copies can return success as long as they are written to the cache layer.

[0003] Compared with centralized caching, distributed multi-layer caching has developed rapidly in recent years, completely breaking the limitation that cache high-speed media and data low-speed disks must be bound together, providing more stable cache performance. The cache can be distributed on several independent servers, or it can share servers with the data layer, and they can use different redundancy strategies, such as 3 copies of the cache server and 2 copies of the data server or others. Through layered caching, the needs of large clusters and large data centers in distributed storage can be better met.

[0004] However, the distributed hierarchical cache architecture in the related art has the following problems:

[0005] 1. When renaming a snapshot, the rename command needs to be sent to multiple layers (cache layer, data layer) at the same time. It is a very complicated process to ensure the transactional nature of the rename in the cache layer and the data layer. In addition, the current data layer is generally composed of low-speed disks, so the rename process will be much slower, which will result in a very high overall rename latency.

[0006] 2. The deletion process is similar to the rename process, and the performance and transaction requirements are very difficult to meet.

[0007] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0008] The embodiments of the present invention provide a data processing method and device based on multi-layer cache, and an electronic device, so as to at least solve the technical problem of complex transaction processing when using a layered cache architecture for data caching in the related technology.

[0009] According to one aspect of an embodiment of the present invention, a data processing method based on multi-layer cache is provided, which is applied to a placement group PG, wherein the placement groups are respectively connected to a cache layer and a data layer, and comprises: receiving a data write request sent by a client, wherein the data write request carries at least: target data to be written; in response to the data write request, generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata; migrating the cache object into the cache layer, and establishing a first mapping relationship between the object metadata of the cache object and the unique identification code; storing the data object in the data layer, and establishing a second mapping relationship between the object metadata of the data object and the unique identification code.

[0010] Optionally, after establishing the first mapping relationship between the object metadata of the cache object and the unique identification code, it also includes: when the object metadata in the cache layer is updated, adjusting the first mapping relationship between the object metadata in the cache layer and the unique identification code.

[0011] Optionally, after establishing the second mapping relationship between the data object and the unique identification code, it also includes: when the cache layer triggers a preset flushing strategy, flushing the cache object from the cache layer to the data object of the data layer; deleting the cache object in the cache layer and retaining the object metadata of the cache object; pointing the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code.

[0012] Optionally, after the object metadata of the cache object in the cache layer is pointed to the data object in the data layer through the unique identification code, it also includes: when it is determined that the cache space in the cache layer meets the space free requirement, releasing the deleted cache object as a read cache; when it is determined that the cache space does not meet the space free requirement, releasing the cache space corresponding to the cache object.

[0013] Optionally, after releasing the cache space corresponding to the cache object, it also includes: when receiving a data read request, searching for the object metadata of the data object through the object metadata of the cache object; querying the data object based on the found object metadata of the data object; and caching the data object in the cache layer.

[0014] Optionally, after generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, it also includes: storing the cache object to the cache layer and the data layer respectively; upon receiving an object deletion instruction, deleting the cache object of the cache layer and updating the object metadata of the cache object; receiving a recycle and flush instruction initiated by a background thread of the cache layer; in response to the recycle and flush instruction, sending an object deletion instruction to the data layer to delete the cache object in the data layer and update the object metadata of the cache object in the data layer; releasing the cache space of the cache object in the cache layer and the cache space of the cache object in the data layer.

[0015] Optionally, in response to the data write request, the step of generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object includes: in response to the data write request, creating a cache object corresponding to the target data; and using an identification allocator to assign the unique identification code to the cache object.

[0016] According to another aspect of an embodiment of the present invention, a data processing device based on a multi-layer cache is provided, which is applied to a placement group PG, wherein the placement groups are respectively connected to a cache layer and a data layer, and include: a receiving unit, used to receive a data write request sent by a client, wherein the data write request carries at least: target data to be written; a responding unit, used to respond to the data write request, generate a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata; a first establishing unit, used to migrate the cache object into the cache layer, and establish a first mapping relationship between the object metadata of the cache object and the unique identification code; a second establishing unit, used to store the data object in the data layer, and establish a second mapping relationship between the object metadata of the data object and the unique identification code.

[0017] Optionally, the data processing device based on multi-layer cache also includes: a first adjustment unit, which is used to adjust the first mapping relationship between the object metadata of the cache object and the unique identification code in the cache layer when the object metadata in the cache layer is updated after establishing the first mapping relationship between the object metadata of the cache object and the unique identification code.

[0018] Optionally, the data processing device based on multi-layer cache also includes: a flushing unit, which is used to flush the cache object from the cache layer to the data object in the data layer after establishing a second mapping relationship between the data object and the unique identification code, when the cache layer triggers a preset flushing strategy; a first deletion unit, which is used to delete the cache object in the cache layer and retain the object metadata of the cache object; and a pointing unit, which is used to point the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code.

[0019] Optionally, the data processing device based on multi-layer cache also includes: a first release unit, which is used to release the deleted cache object as a read cache after pointing the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code, if it is determined that the cache space in the cache layer meets the space free requirement; and a second release unit, which is used to release the cache space corresponding to the cache object if it is determined that the cache space does not meet the space free requirement.

[0020] Optionally, the data processing device based on multi-layer cache also includes: a first search unit, used to search for the object metadata of the data object through the object metadata of the cache object when a data read request is received after the cache space corresponding to the cache object is released; a second search unit, used to query the data object based on the object metadata of the data object found; and a cache unit, used to cache the data object in the cache layer.

[0021] Optionally, the data processing device based on multi-layer cache also includes: a first storage unit, which is used to store the cache objects to the cache layer and the data layer respectively after generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object; a second deletion unit, which is used to delete the cache object of the cache layer and update the object metadata of the cache object when receiving an object deletion instruction; a first receiving module, which is used to receive a recycling and flushing instruction initiated by the background thread of the cache layer; a first response module, which is used to respond to the recycling and flushing instruction and send an object deletion instruction to the data layer to delete the cache object in the data layer and update the object metadata of the cache object in the data layer; a first release module, which is used to release the cache space of the cache object of the cache layer and the cache space of the cache object in the data layer.

[0022] Optionally, the response unit includes: a second response module, used to respond to the data write request and create a cache object corresponding to the target data; and an allocation module, used to use an identification allocator to allocate the unique identification code to the cache object.

[0023] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned multi-layer cache-based data processing methods by executing the executable instructions.

[0024] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned multi-layer cache-based data processing methods.

[0025] In the present application, after receiving a data write request sent by a client, a data write request is responded to, a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object are generated, wherein the cache object and the data object respectively correspond to object metadata, the cache object is moved into the cache layer, and a first mapping relationship between the object metadata of the cache object and the unique identification code is established, and the data object is stored in the data layer, and a second mapping relationship between the object metadata of the data object and the unique identification code is established. In this embodiment, a unique identification code is used to associate the object mapping relationship in the data layer and the cache layer, which reduces the complexity of the transactional processing method, and ensures the correspondence between the cache object and the data object, which can improve the data query efficiency and the cache efficiency, thereby solving the technical problem of complex transactional processing methods when a layered cache architecture is used for data caching in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 is a flowchart of an optional data processing method based on multi-layer cache according to an embodiment of the present invention;

[0028] Figure 2 is an object mapping relationship diagram of an optional hierarchical cache according to an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of data flushing of an optional hierarchical cache according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of an optional data processing device based on multi-layer cache according to an embodiment of the present invention;

[0031] Figure 5 The present invention is a hardware structure block diagram of an electronic device (or mobile device) according to a data processing method based on a multi-layer cache according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] To facilitate those skilled in the art to understand the present invention, some terms or nouns appearing in the embodiments of the present invention are explained below:

[0035] Placement Group, PG for short, implements policy isolation between different storage pools and is used to organize and map the storage of objects.

[0036] Object, Object, the smallest data storage unit managed by the cluster.

[0037] The data layer, also known as the Data Pool, is the data layer in the storage hierarchy and is composed of relatively low-speed storage media such as mechanical disks HDDs and ordinary solid-state disks SSDs in each node.

[0038] The cache layer, also known as Cache OSD, Cache, is the cache layer in the storage hierarchy and consists of high-speed storage media such as SSD and NVMe on each node.

[0039] Data volume, volume, referred to as volume, logical storage space;

[0040] Storage bucket, buckut, can also be called cache bucket.

[0041] The present invention can be applied to various data storage systems\cache devices, which include: a client, a cache layer (or a cache pool Cache), and a data layer (or a data storage pool DataPool). Through the data storage system\cache device, data snapshot backup, data flushing, data elimination, etc. can be achieved. For data snapshot processing, the ROW method is mainly implemented, and it is achieved by renaming the snapshot, creating new metadata and establishing a new mapping relationship each time a snapshot is generated. When the snapshot is renamed, when the object metadata information in the high-speed medium is updated, the generated snapshot needs to point to the source data object, and a series of data updates will be performed.

[0042] In the present invention, an object mapping method and processing strategy between multiple layers of cache are introduced, including the establishment and management of mapping relationships, the uniqueness of mappings, and the use of mapping relationships to process data flushing and space recovery, etc., which can solve the problems faced by the hierarchical structure, including snapshot rename or other metadata updates in the hierarchical structure. While ensuring uniqueness, the implementation scheme is simpler and the transaction processing latency is significantly reduced.

[0043] The present invention can provide support conditions for asynchronous background data processing between multiple layers, avoiding the performance degradation caused by the synchronous processing method, and can effectively solve the data consistency problems that may be caused by various abnormal switches in the multi-layer linkage process.

[0044] The following embodiments of the present invention can implement a scheme of object mapping, allocation and deletion mechanism of layered cache. The following is an explanation of the embodiments of the present invention.

[0045] Embodiment 1

[0046] According to an embodiment of the present invention, an embodiment of a data processing method based on multi-layer cache is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] The embodiment of the present invention provides a data processing method based on multi-layer cache, which is applied to a placement group PG, and the placement group is connected to a cache layer and a data layer respectively. The PG in this embodiment is responsible for receiving and processing requests from clients; and is responsible for translating these data requests into transactions that can be understood by local object storage.

[0048] Figure 1 is a flowchart of an optional data processing method based on multi-layer cache according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0049] Step S102: receiving a data writing request sent by a client, wherein the data writing request carries at least target data to be written.

[0050] In the embodiment of the present invention, the types of clients include but are not limited to: PC, notebook, mobile terminal, panel, etc. The client may refer to the business layer or the request initiation layer, and when data needs to be stored or retrieved, a request needs to be sent to the cache layer.

[0051] Step S104, in response to the data write request, generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata.

[0052] Optionally, in response to a data write request, the steps of generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object include: in response to the data write request, creating a cache object corresponding to the target data; using an identification allocator to assign a unique identification code to the cache object.

[0053] The unique identification code in this embodiment may refer to UUID. The number of digits of the unique identification code is set according to different devices. When data IO is written for the first time, a new cache object A can be generated, and a data object A* is generated at the same time, and it will be persisted. In this embodiment, each time an object is generated, a UUID (universally unique identifier) ​​will be applied for, and the natural uniqueness of A* is guaranteed. The name space range of the UUID is within the placement group PG, and the UUID corresponds to the object in the data layer.

[0054] Figure 2 is an object mapping relationship diagram of an optional hierarchical cache according to an embodiment of the present invention, such as Figure 2 As shown, it includes: a storage cache layer (pointing to A), a 64-bit UUID mapping, and a storage data layer (pointing to A*).

[0055] A UUID is used to associate the object mapping relationship between the data layer and the cache layer to implement transactional processing. When adding an object, the identifier distributor branch will be entered. The allocation of a UUID is based on the PG namespace to reduce the probability of errors.

[0056] Optionally, in this embodiment, each time a snapshot is generated, new metadata is created and a new mapping relationship is established. When the snapshot is renamed and the object metadata information in the high-speed medium is updated, the generated snapshot needs to point to the source data object, and a series of data updates will be performed.

[0057] Step S106: Migrate the cache object into the cache layer, and establish a first mapping relationship between the object metadata of the cache object and the unique identification code.

[0058] Step S108: store the data object in the data layer, and establish a second mapping relationship between the object metadata of the data object and the unique identification code.

[0059] Through the above steps, after receiving the data write request sent by the client, the data write request can be responded to, and a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object can be generated, wherein the cache object and the data object respectively correspond to object metadata, the cache object is moved into the cache layer, and a first mapping relationship between the object metadata of the cache object and the unique identification code is established, and the data object is stored in the data layer, and a second mapping relationship between the object metadata of the data object and the unique identification code is established. In this embodiment, a unique identification code is used to associate the object mapping relationship in the data layer and the cache layer, which reduces the complexity of the transactional processing method and ensures the correspondence between the cache object and the data object, which can improve the data query efficiency and the cache efficiency, thereby solving the technical problem of complex transactional processing methods when using a layered cache architecture for data caching in related technologies.

[0060] The embodiments of the present invention are described in detail below in combination with the above-mentioned implementation steps.

[0061] As an optional implementation of this embodiment, after establishing the first mapping relationship between the object metadata of the cache object and the unique identification code, it also includes: when the object metadata in the cache layer is updated, adjusting the first mapping relationship between the object metadata in the cache layer and the unique identification code.

[0062] When the object metadata of the cache layer is updated, such as when the snapshot policy is triggered to produce a ROW snapshot, the object metadata in the cache layer will be modified. At this time, it is necessary to synchronously modify its mapping relationship with the unique identification code to ensure the correspondence between the metadata object and the object in the data layer.

[0063] In this embodiment, after the object metadata in the cache layer is updated, its mapping relationship with the unique identification code is modified. This can ensure the correspondence between the cache object and the data object. When it cannot be obtained in the cache, it can be quickly and accurately searched in the data layer, which improves efficiency and ensures relevance after the hot data is promoted.

[0064] Optionally, after establishing the second mapping relationship between the data object and the unique identification code, it also includes: when the cache layer triggers a preset flushing strategy, flushing the cache object from the cache layer to the data object in the data layer; deleting the cache object in the cache layer and retaining the object metadata of the cache object; pointing the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code.

[0065] The cache layer / Cache layer caches the metadata of all cached objects, including all data objects. When the cache layer triggers certain policies, such as water level policy and idle data flushing policy, the cache objects in the cache layer will be deleted and the object metadata will be retained.

[0066] Figure 3 FIG. 1 is a schematic diagram of data flushing of an optional hierarchical cache according to an embodiment of the present invention. Figure 3 As shown in the figure, when the cache layer triggers certain strategies, such as water level strategy and idle data flushing strategy, a data object A (object A) is flushed from the cache layer to the data object A* (object A*) of the data layer; the cache layer data object A will be deleted, but the object metadata A will be stored all the time; the object metadata A in the cache layer points to the data object object A* in the data layer through UUID.

[0067] Another optional method is to, after the object metadata of the cache object in the cache layer is pointed to the data object in the data layer through a unique identification code, further include: when it is determined that the cache space in the cache layer meets the space free requirement, releasing the deleted cache object as a read cache; when it is determined that the cache space does not meet the space free requirement, releasing the cache space corresponding to the cache object.

[0068] In this embodiment, after the cache space corresponding to the cache object is released, it also includes: when a data read request is received, searching for the object metadata of the data object through the object metadata of the cache object; querying the data object based on the object metadata of the found data object; and caching the data object in the cache layer.

[0069] Combination Figure 3, the deleted object A will be released as a read cache. When there is data to read, it can be returned from objectA. When the cache space is insufficient, the cache space with lower hot spots will be released. When the space is released, when the data needs to be read again, it is necessary to find object metadata A* through object metadata A, and then find object A*, promote it to the cache layer, and then access it by the front-end application.

[0070] Optionally, after generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, it also includes: storing the cache object in the cache layer and the data layer respectively; upon receiving an object deletion instruction, deleting the cache object in the cache layer, and updating the object metadata of the cache object; receiving a recycling and flushing instruction initiated by the background thread of the cache layer; in response to the recycling and flushing instruction, sending an object deletion instruction to the data layer to delete the cache object in the data layer, and update the object metadata of the cache object in the data layer; releasing the cache space of the cache object in the cache layer and the cache space of the cache object in the data layer.

[0071] The number of objects (cached objects) in the data layer determines the available space in the entire pool. The cache layer is an intermediate process. Data objects usually have two states, in the cache layer or in the data layer. In some special cases, data objects can exist in both the cache layer and the data layer. In this case, in the process of deleting data objects, cache objects in both the cache layer and the data layer need to be cleared at the same time.

[0072] The cache layer caches all object metadata. After receiving the command to delete the cached object, it first deletes the cached object in the cache layer (if it exists), updates the object metadata (Meta data), and completes the deletion operation of the cached object in the cache layer; the cache OSD background thread initiates a recycling flush operation (if necessary), then sends a deletion command to the data layer OSD, waits for the return processing, deletes the corresponding object, updates the Meta data, completes the operation on the data layer, and truly releases all space.

[0073] In this embodiment, the above strategy requires the complete uniqueness of the data object A* corresponding to the cache object A. If A is deleted and then created, the corresponding A* should be regenerated.

[0074] In this embodiment, under abnormal circumstances, data objects may exist in both the cache layer and the data layer. In this case, in the process of deleting the data object, it is necessary to clear the objects in both the cache layer and the data layer at the same time. The recycling mechanism can recycle the cache space well to avoid space occupation.

[0075] The present invention is described below in conjunction with another optional embodiment.

[0076] Embodiment 2

[0077] The embodiment of the present invention provides a data processing device based on multi-layer cache, which is applied to a placement group PG, and the placement group is connected to a cache layer and a data layer respectively. The data processing device based on multi-layer cache includes multiple implementation units corresponding to the various implementation steps in the above embodiment 1.

[0078] Figure 4 is a schematic diagram of an optional data processing device based on multi-layer cache according to an embodiment of the present invention, such as Figure 4 As shown, it includes: a receiving unit 41, a responding unit 43, a first establishing unit 45, and a second establishing unit 47, wherein:

[0079] The receiving unit 41 is used to receive a data writing request sent by a client, wherein the data writing request carries at least: target data to be written;

[0080] A response unit 43 is used to respond to the data write request, generate a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata;

[0081] A first establishing unit 45, configured to migrate the cache object into the cache layer and establish a first mapping relationship between the object metadata of the cache object and the unique identification code;

[0082] The second establishing unit 47 is used to store the data object in the data layer and establish a second mapping relationship between the object metadata of the data object and the unique identification code.

[0083] The above-mentioned data processing device based on multi-layer cache can, after receiving the data write request sent by the client through the receiving unit 41, respond to the data write request through the responding unit 43, generate a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata, migrate the cache object into the cache layer through the first establishing unit 45, and establish a first mapping relationship between the object metadata of the cache object and the unique identification code, and store the data object in the data layer through the second establishing unit 47, and establish a second mapping relationship between the object metadata of the data object and the unique identification code. In this embodiment, a unique identification code is used to associate the object mapping relationship in the data layer and the cache layer, reduce the complexity of the transactional processing method, and ensure the corresponding relationship between the cache object and the data object, which can improve the data query efficiency and cache efficiency, thereby solving the technical problem of complex transactional processing methods when using a layered cache architecture for data caching in related technologies.

[0084] Optionally, the data processing device based on multi-layer cache also includes: a first adjustment unit, which is used to adjust the first mapping relationship between the object metadata of the cache object and the unique identification code in the cache layer when the object metadata in the cache layer is updated after establishing the first mapping relationship between the object metadata of the cache object and the unique identification code.

[0085] Optionally, the data processing device based on multi-layer cache also includes: a flushing unit, which is used to flush the cache object from the cache layer to the data object in the data layer after establishing a second mapping relationship between the data object and the unique identification code, when the cache layer triggers a preset flushing strategy; a first deletion unit, which is used to delete the cache object in the cache layer and retain the object metadata of the cache object; and a pointing unit, which is used to point the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code.

[0086] Optionally, the data processing device based on multi-layer cache also includes: a first release unit, which is used to release the deleted cache object as a read cache after pointing the object metadata of the cache object in the cache layer to the data object in the data layer through a unique identification code, and when it is determined that the cache space in the cache layer meets the space free requirement; a second release unit, which is used to release the cache space corresponding to the cache object when it is determined that the cache space does not meet the space free requirement.

[0087] Optionally, the data processing device based on multi-layer cache also includes: a first search unit, used to search for object metadata of the data object through the object metadata of the cache object when a data read request is received after the cache space corresponding to the cache object is released; a second search unit, used to query the data object based on the object metadata of the found data object; and a cache unit, used to cache the data object in the cache layer.

[0088] Optionally, the data processing device based on multi-layer cache also includes: a first storage unit, which is used to store the cache objects in the cache layer and the data layer respectively after generating the cache objects corresponding to the target data and the data objects and unique identification codes corresponding to the cache objects; a second deletion unit, which is used to delete the cache objects in the cache layer and update the object metadata of the cache objects when receiving the object deletion instruction; a first receiving module, which is used to receive the recycling and flushing instructions initiated by the background thread of the cache layer; a first response module, which is used to respond to the recycling and flushing instructions and send the object deletion instructions to the data layer to delete the cache objects in the data layer and update the object metadata of the cache objects in the data layer; a first release module, which is used to release the cache space of the cache objects in the cache layer and the cache space of the cache objects in the data layer.

[0089] Optionally, the response unit includes: a second response module, used to respond to the data write request and create a cache object corresponding to the target data; and an allocation module, used to use an identification allocator to allocate a unique identification code to the cache object.

[0090] The above-mentioned data processing device based on multi-layer cache can also include a processor and a memory. The above-mentioned receiving unit 41, response unit 43, first establishment unit 45, second establishment unit 47, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0091] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. The kernel can be set to one or more, and responds to data write requests by adjusting kernel parameters, generates cache objects corresponding to target data and data objects and unique identification codes corresponding to the cache objects, migrates the cache objects to the cache layer, and establishes a first mapping relationship between the object metadata of the cache objects and the unique identification code, stores the data objects in the data layer, and establishes a second mapping relationship between the object metadata of the data objects and the unique identification code.

[0092] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0093] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned multi-layer cache-based data processing methods by executing the executable instructions.

[0094] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned multi-layer cache-based data processing methods.

[0095] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: receiving a data write request sent by a client, wherein the data write request carries at least: target data to be written; responding to the data write request, generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata; migrating the cache object into a cache layer, and establishing a first mapping relationship between the object metadata of the cache object and the unique identification code; storing the data object in the data layer, and establishing a second mapping relationship between the object metadata of the data object and the unique identification code.

[0096] Figure 5 1 is a hardware structure block diagram of an electronic device (or mobile device) according to a data processing method based on multi-layer cache according to an embodiment of the present invention. Figure 5 As shown, the electronic device may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations are shown.

[0097] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method based on multi-layer cache, characterized in that: Applied to the placement group PG, which is connected to the cache layer and the data layer respectively, including: Receiving a data writing request sent by a client, wherein the data writing request carries at least: target data to be written; In response to the data write request, a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object are generated, wherein the cache object and the data object respectively correspond to object metadata; Migrating the cache object into the cache layer, and establishing a first mapping relationship between the object metadata of the cache object and the unique identification code; Storing the data object in a data layer, and establishing a second mapping relationship between the object metadata of the data object and the unique identification code; In the case where the cache layer triggers a preset flushing strategy, flush the cache object from the cache layer to the data object of the data layer; delete the cache object in the cache layer and retain the object metadata of the cache object; point the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code; When it is determined that the cache space does not meet the space free requirement, releasing the cache space corresponding to the cache object; In case of receiving a data read request, searching for the object metadata of the data object through the object metadata of the cache object; querying the data object based on the found object metadata of the data object; and caching the data object in the cache layer.

2. The data processing method according to claim 1, characterized in that: After establishing a first mapping relationship between the object metadata of the cache object and the unique identification code, the method further includes: When the object metadata in the cache layer is updated, the first mapping relationship between the object metadata in the cache layer and the unique identification code is adjusted.

3. The data processing method according to claim 1, characterized in that: After the object metadata of the cache object in the cache layer is pointed to the data object in the data layer through the unique identification code, the method further includes: When it is determined that the cache space in the cache layer meets the space free requirement, the deleted cache object is released as a read cache.

4. The data processing method according to claim 1, characterized in that: After generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, the method further includes: Storing the cache objects in the cache layer and the data layer respectively; In case of receiving an object deletion instruction, deleting the cache object of the cache layer and updating the object metadata of the cache object; Receive a recycle and flush instruction initiated by a background thread of the cache layer; In response to the reclaim and flush instruction, an object deletion instruction is sent to the data layer to delete the cached object in the data layer and update the object metadata of the cached object in the data layer; The cache space of the cache object in the cache layer and the cache space of the cache object in the data layer are released.

5. The data processing method according to claim 1, characterized in that: In response to the data write request, the step of generating a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object comprises: In response to the data write request, create a cache object corresponding to the target data; An identification allocator is used to allocate the unique identification code to the cache object.

6. A data processing device based on multi-layer cache, characterized in that: Applied to the placement group PG, which is connected to the cache layer and the data layer respectively, including: A receiving unit, configured to receive a data writing request sent by a client, wherein the data writing request carries at least: target data to be written; A response unit, configured to respond to the data write request, generate a cache object corresponding to the target data and a data object and a unique identification code corresponding to the cache object, wherein the cache object and the data object respectively correspond to object metadata; A first establishing unit, configured to migrate the cache object into the cache layer and establish a first mapping relationship between the object metadata of the cache object and the unique identification code; A second establishing unit, configured to store the data object in a data layer and establish a second mapping relationship between the object metadata of the data object and the unique identification code; The data processing device based on multi-layer cache further includes: a flushing unit, which is used to flush the cache object from the cache layer to the data object of the data layer when the cache layer triggers a preset flushing strategy after establishing the second mapping relationship between the data object and the unique identification code; a first deleting unit, which is used to delete the cache object in the cache layer and retain the object metadata of the cache object; a pointing unit, which is used to point the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code; The first release unit is used to release the deleted cache object as a read cache after pointing the object metadata of the cache object in the cache layer to the data object in the data layer through the unique identification code, if it is determined that the cache space in the cache layer meets the space free requirement; the second release unit is used to release the cache space corresponding to the cache object if it is determined that the cache space does not meet the space free requirement; The first search unit is used to search for the object metadata of the data object through the object metadata of the cache object when a data read request is received after the cache space corresponding to the cache object is released; the second search unit is used to query the data object based on the object metadata of the data object found; the cache unit is used to cache the data object in the cache layer.

7. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the multi-layer cache-based data processing method described in any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-layer cache-based data processing method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • method for realizing CEPH cluster caching and an OSD

    CN109799954A

  • Data management system and data management method

    US20160006829A1