Multi-level cache method, device, computer equipment and storage medium

Abstract

The present application relates to a multi-level cache method, device, computer equipment and storage medium, which are applied to a cloud storage system, wherein the cloud storage system includes a client and several storage nodes, and the storage nodes include a primary cache and a secondary cache; wherein the method divides the target data in the client into several slice data; the slice data is stored in the form of stripes; when the storage node receives a first write request for the target data, the stripes in the client are merged and written to the primary cache; when the storage node receives a second write request for the target data, the complete slice data of each target data in the primary cache is written to the secondary cache. Through the present application, the merged stripes in the client can be written to the primary cache of the storage node, and the primary cache can be written to the secondary cache after receiving the complete slice data of each target data, thereby reducing small IO operations and solving the problem of a large number of small-scale IO operations affecting the overall performance of the system.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a multi-level cache method, apparatus, computer equipment and storage medium. Background Art

[0002] Distributed cloud storage architecture is a storage architecture model based on distributed computing and cloud computing concepts. It stores data in multiple storage servers and forms a virtual storage device through the network. This architecture model aims to achieve efficient, scalable, highly available and reliable data storage services.

[0003] When the distributed cloud storage architecture processes image and small file services, since images and small files usually have a small amount of data, a large number of small-scale IO (Input / Output) operations will be generated when storing and accessing these files, increasing network interactions and reducing storage write performance, affecting the overall performance of the system.

[0004] There is a large number of small-scale IO operations in related technologies, which affects the overall performance of the system. Currently, no effective solution has been proposed. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-level cache method, device, computer equipment and storage medium that can reduce small-scale IO operations in response to the above technical problems.

[0006] In a first aspect, a multi-level cache method is provided in this embodiment, which is applied to a cloud storage system, wherein the cloud storage system includes a client and a plurality of storage nodes, wherein the storage nodes include a first-level cache and a second-level cache; the method includes:

[0007] Dividing the target data in the client into a plurality of slice data; storing the slice data in the form of stripes;

[0008] When the storage node receives the first write request for the target data, the storage node merges the stripes in the client and writes them into the first-level cache;

[0009] When the storage node receives a second write request for the target data, the storage node writes complete slice data of each target data in the first-level cache into the second-level cache.

[0010] In some embodiments, merging the stripes in the client and writing them into the first-level cache includes:

[0011] The stripes satisfying a preset merging rule are merged; the preset merging rule is that the stripes belong to the same target data and the memory addresses of the stripes are continuous.

[0012] In some of the embodiments, it also includes:

[0013] According to the response of the storage node to the write request of the client, the storage node is uniformly scheduled and managed through the corresponding asynchronous thread in the client.

[0014] In some embodiments, the step of merging the stripes in the client and writing them into the first-level cache further includes:

[0015] According to the status of the slice data in the storage node, a corresponding strategy is adopted to update the status; and a first response to the first write request is returned to the client.

[0016] In some of the embodiments, it also includes:

[0017] When the client receives the first response of the storage node to the first write request, the client stores the target data information corresponding to the slice data in the completed state into the submission queue;

[0018] In the submission queue, a second write request corresponding to the target data is sent to the storage node through a submission thread.

[0019] In some of the embodiments, it also includes:

[0020] When the client receives the second response of the storage node to the second write request, the client stores the target data information corresponding to the successfully written slice data in the migration queue;

[0021] In the migration queue, a migration request corresponding to the target data is sent to the storage node through a migration data thread.

[0022] In some of the embodiments, it also includes:

[0023] When the storage node receives a migration request for the target data, each complete slice data of the target data in the secondary cache is migrated to the HDD of the storage node.

[0024] In some of the embodiments, it also includes:

[0025] When the client receives the third response of the storage node to the migration request, the target data information corresponding to the successfully written slice data is stored in the deletion queue;

[0026] In the deletion queue, a deletion request corresponding to the target data is sent to the secondary cache through a deletion thread.

[0027] In a second aspect, a multi-level cache device is provided in this embodiment, which is applied to a cloud storage system, wherein the cloud storage system includes a client and a plurality of storage nodes, wherein the storage nodes include a first-level cache and a second-level cache; the device includes:

[0028] A data slicing module, used for dividing the target data in the client into a plurality of slice data; the slice data is stored in the form of stripes;

[0029] A first cache module, configured to merge the stripes in the client and write them into the first-level cache when the storage node receives a first write request for the target data;

[0030] The second cache module is used to write the complete slice data of each target data in the first-level cache into the second-level cache when the storage node receives a second write request for the target data.

[0031] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-level cache method described in the first aspect when executing the computer program.

[0032] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the multi-level cache method described in the first aspect is implemented.

[0033] Compared with the related art, the multi-level cache method, device, computer equipment and storage medium provided in this embodiment are applied to a cloud storage system, wherein the cloud storage system includes a client and several storage nodes, wherein the storage nodes include a primary cache and a secondary cache; by dividing the target data in the client into several slice data; the slice data is stored in the form of stripes; when the storage node receives a first write request for the target data, the stripes in the client are merged and written to the primary cache; when the storage node receives a second write request for the target data, the complete slice data of each target data in the primary cache is written to the secondary cache. Through this embodiment, the merged stripes in the client can be written to the primary cache of the storage node, and the primary cache can be written to the secondary cache after receiving the complete slice data of each target data, thereby reducing small IO operations and solving the problem of a large number of small-scale IO operations affecting the overall performance of the system.

[0034] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 is a hardware structure block diagram of a terminal of a multi-level cache method in an embodiment;

[0037] Figure 2 is a flowchart of a multi-level cache method in an embodiment;

[0038] Figure 3 is a schematic diagram of dividing target data in a client in an embodiment;

[0039] Figure 4 is a schematic diagram of a client writing target data to a storage node in an embodiment;

[0040] Figure 5 is a schematic diagram of a cloud storage system in one embodiment;

[0041] Figure 6 is a schematic diagram of a process of issuing and executing a second write request in an embodiment;

[0042] Figure 7 is a schematic diagram of a process of issuing and executing a migration request in an embodiment;

[0043] Figure 8 is a schematic diagram of the process of issuing and executing a deletion request in an embodiment;

[0044] Fig. 9 is a flowchart of a multi-level caching method in another embodiment;

[0045] Fig.10 The structure block diagram of a multi-level cache device in an embodiment.

[0046] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, data slicing module; 20, first cache module; 30, second cache module. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the multi-level cache method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the multi-level cache method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0052] Distributed cloud storage architecture is a storage architecture model based on distributed computing and cloud computing concepts. It stores data in multiple storage servers and forms a virtual storage device through the network. This architecture model aims to achieve efficient, scalable, highly available and reliable data storage services.

[0053] When the distributed cloud storage architecture processes image and small file services, since images and small files usually have a small amount of data, a large number of small-scale IO (Input / Output) operations will be generated when storing and accessing these files, increasing network interactions and reducing storage write performance, affecting the overall performance of the system.

[0054] In this embodiment, a multi-level cache method is provided. The method is applied to a cloud storage system. The cloud storage system includes a client and several storage nodes. The storage nodes include a first-level cache and a second-level cache.

[0055] Among them, the client connects to the metadata service, and the client interacts with the metadata service to create files, apply for objects, open target data, return target data ID, storage node ID and other related operations.

[0056] The client establishes connections with multiple storage nodes, which are used to store real physical data. Different asynchronous threads in the client perform unified scheduling and management of the connected storage nodes. The storage node includes a cache layer and a HDD (Hard Disk Drive). The cache layer includes memory and SSD (Solid State Drive). The memory and SSD serve as the first-level cache and second-level cache of the storage node respectively.

[0057] Figure 2 is a flow chart of the multi-level cache method in this embodiment. Figure 2 As shown, the method comprises the following steps:

[0058] Step S210, dividing the target data in the client into a plurality of slice data; the slice data is stored in the form of stripes.

[0059] Specifically, a block of memory space is applied for in the client, and the memory space is initialized into multiple stripes. When the client writes the target data, the target data is divided into several slices of data based on the preset EC (Erasure Coding) redundancy rules, and each slice of data is stored in the form of stripes. Among them, the target data can be image data or small files in the video cloud storage, etc. The target data is segmented and encoded through the encoding algorithm to generate redundant data blocks. When the original target data is lost or damaged, the data can be recovered through these redundant data blocks.

[0060] According to the EC redundancy rule, slice data includes data slices and check data slices. Each slice data is further divided into smaller data blocks (pieces). Striping is a technology that stores multiple data blocks on different physical storage devices. Each data block is regarded as a part of a stripe, and each stripe includes the corresponding data blocks in each slice data.

[0061] Take the "3+1" EC redundancy rule as an example (for each target data, 3 data slices and 1 additional verification data slice will be retained). Figure 3 Schematic diagram of dividing target data in the client in this embodiment. Figure 3 As shown in the figure, it is assumed that the size of each data slice is 1 megabyte (M), the size of 3 data slices is 3 megabytes, the size of the check data slice is 1 megabyte, each slice data is composed of N data blocks (piece), the data blocks (piece) at the corresponding position of each slice data form a stripe, each stripe is composed of 4 data blocks (piece), a total of N stripes, the slice data in the client is stored in the form of stripes, and when writing slice data, the stripe is used as the basic unit, and the data blocks (piece) in each stripe are written to the corresponding storage node.

[0062] When the client writes the target data to the storage node, it first creates a file through the metadata service (MDS). The metadata service returns the target data ID and the storage node ID corresponding to the slice data of the target data. The client first writes the slice data to the client's memory, where it needs to be cached multiple times in the form of stripes. Furthermore, the stripes can be merged in the client to obtain a full stripe, and the data blocks (pieces) in the full stripe are written to the corresponding storage node.

[0063] Step S220: When the storage node receives the first write request for the target data, it merges the stripes in the client and writes them into the primary cache.

[0064] Specifically, when the storage node starts, it applies for a piece of memory space as the first-level cache. The first-level cache can store slices of multiple target data, and its size can be calculated and determined based on the size (Size) of the slice data and the number (Count) of the target data (Size×Count). In the first-level cache, a slice object is initialized for each slice data, and the slice object points to the memory space of the storage node.

[0065] Three sets for slice objects need to be maintained in the storage node, each set represents a state of the slice object, namely: slice objects in the state of being written belong to the writing set using_piece_map, slice objects in the initial state belong to the initial set idle_piece_list, and slice objects in the state of completed belong to the finished set finish_piece_map. Among them, slice objects in the state of being written indicate that the corresponding slice data is being written multiple times; slice objects in the state of completed indicate that the pieces of the corresponding slice data have been written completely; all unused slice objects, that is, the reinitialized pieces are in the initial state, and are sorted in the initial set idle_piece_list according to the address order in the memory.

[0066] The slice object stores the data information of each slice data, including the information of the target data to which the slice data belongs (such as ID, name), write offset, slice object cache space, status, written data length, and write path, etc.

[0067] The field names and corresponding field descriptions in the data information are shown in Table 1 below.

[0068] Table 1:

[0069]

[0070] The first write request indicates a request to write the slice data of the target data from the client to the first-level cache of the storage node. When the storage node receives the first write request from the client, for the merged stripes in the client, each piece in the stripe is written to the corresponding storage node; for the single stripe that has not been merged in the client, each piece in the single stripe is also written to the corresponding storage node. After completing the first write request, the storage node will return the first response to the client, which includes the data length and status of the slice data written this time.

[0071] Step S230, when the storage node receives the second write request for the target data, the complete slice data of each target data in the first-level cache is written into the second-level cache.

[0072] Specifically, after receiving the slice data, the storage node first caches it in the first-level cache. The second write request indicates a request to write the slice data from the first-level cache to the second-level cache. When the storage node receives the second write request from the client, after the slice data is completely received, it will determine the target data that has been written to the complete slice data based on the completion status of the slice data, and at the same time, according to the load balancing strategy, determine the write path of the slice data, that is, the specific second-level cache SSD and HDD to be written, and update the path information ssdPath and hddPath in the above data information accordingly. After the slice data is successfully written to the second-level cache, the storage node will return a second response of successful write to the client.

[0073] Through the above steps, the merged stripes in the client can be written into the first-level cache of the storage node, and then written into the second-level cache after the first-level cache receives the complete slice data of each target data, thereby reducing the network interaction and small IO operations between the client and the storage node, improving the reading and writing efficiency, and solving the problem of a large number of small-scale IO operations affecting the overall performance of the system.

[0074] In some of the embodiments, the following steps are also included:

[0075] In the client, the stripes that meet the preset merging rule are merged; the preset merging rule is that the stripes belong to the same target data and the memory addresses of the stripes are continuous.

[0076] Specifically, the slice data is written to the client's memory, where for one target data, multiple write caches are required, and after each write cache is completed, the stripes that meet the preset merge rule are merged, and then each piece in the merged full stripe is written to the corresponding storage node level 1 cache. The preset merge rule is that the stripes belong to the same target data and the memory addresses of the stripes are continuous. For stripes that do not meet the preset merge rule, each piece in the single stripe is written to the corresponding storage node level 1 cache. In addition, the status of whether the current target data has been fully written, the write position offset, the slice data, the target data ID, the target data name and other information are passed to the storage node.

[0077] Figure 4 FIG. 1 is a schematic diagram of a client writing target data to a storage node in this embodiment. Figure 4 As shown, when the client writes target data to the storage node, it first creates a file through the metadata service (MDS). The metadata service returns the target data ID and the storage node ID corresponding to the slice data of the target data. The client first writes the slice data to the client's memory. For example, the client cache size defaults to 8 stripes, and the data block (piece) size of each stripe is 4K, which means that the cache can only write 96K (4K×3×8) at a time. For target data of 3 megabytes in size, the cache needs to be written multiple times. After each cache write is completed, the stripes with consecutive cache memory addresses are merged, and the merged stripes are written to the corresponding storage nodes according to each data block (piece).

[0078] In this embodiment, by merging stripes in the client and then writing each piece in the merged stripe to the storage node, small IO operations between the client and the storage node can be reduced.

[0079] In some of the embodiments, the step of merging the stripes in the client and writing them into the primary cache further includes the following steps:

[0080] According to the status of the slice data in the storage node, a corresponding strategy is adopted to update the status; and a first response to the first write request is returned to the client.

[0081] Specifically, according to the above description, each storage node maintains three sets for slice objects: slice objects in the state of being written belong to the write set using_piece_map, slice objects in the initial state belong to the initial set idle_piece_list, and slice objects in the state of completed belong to the completed set finish_piece_map. When receiving the first write request from the client, parse the request parameters of the first write request, such as the status and other parameters, and use the corresponding strategy to update the status according to the status of the slice data in the storage node. The specific strategies are as follows:

[0082] (1) If the slice data is in the initial state init, the corresponding slice object is obtained from the initial set idle_piece_list, the status of the slice object is set to writing, and the slice object is added to the writing set using_piece_map. The key-value pair key in the set is the target data ID, and the value is the slice object. At the same time, the write offset offset and the written data length dataLen of the slice object are updated, and the length of the data written this time is returned to the client.

[0083] (2) If the slice data status is writing, the slice object corresponding to the current small IO data (i.e., the piece in the slice data) is determined from the writing set using_piece_map according to the corresponding target data ID, and the slice object cache space buf, write offset offset, and written data length dataLen corresponding to the slice object are updated, and the data length of this write is returned to the client.

[0084] (3) If the slice data status is finished, the slice object corresponding to the current small IO data (i.e., the piece in the slice data) is determined from the write set using_piece_map according to the target data ID, and the slice object cache space buf, write offset offset, and written data length dataLen corresponding to the slice object are updated. At the same time, the slice object status is set to finish, and the slice object is deleted from the write set using_piece_map, added to the finished finish_piece_map, and the length of the data written this time is returned to the client.

[0085] After completing the first write request, the storage node will return the first response to the client. The first response includes information such as the data length, status, target data ID, and target data name of the slice data written this time.

[0086] In some of the embodiments, the following steps are also included:

[0087] According to the storage node's response to the client's write request, the storage node is uniformly scheduled and managed through the corresponding asynchronous thread in the client.

[0088] Specifically, after the storage node completes the client's write request (such as the first write request and the second write request in the above embodiment), it will return a successful write response to the client. Since the storage nodes in the cloud storage system are not interoperable with each other, and multiple slices of a target data are in different storage nodes, based on this, after the client receives the response result in this embodiment, the storage node is uniformly scheduled and managed through the corresponding asynchronous thread. For example, the client uniformly issues write requests, migration requests, deletion requests and other operations to each storage node, detects the response result of each storage node, and performs corresponding processing.

[0089] In this embodiment, unified data flow scheduling and management of storage nodes are performed through multiple asynchronous threads in the client background. It does not rely on other services, can reduce complexity, and ensures the integrity of target data and consistency of data migration. Before all slice data is written to the HDD, the entire data process is in the cache, providing real-time write-and-read capabilities.

[0090] Figure 5 is a schematic diagram of a cloud storage system in an embodiment. Figure 5 As shown, the cloud storage system includes a client and several storage nodes, and the storage nodes include a first-level cache and a second-level cache.

[0091] Among them, the client connects to the metadata service, and the client interacts with the metadata service (MDS) to create files, apply for objects, open target data, and return target data ID, storage node ID and other related operations.

[0092] The client establishes connections with multiple storage nodes, applies for a piece of memory space on the client, and initializes the space into multiple stripes. After each cache write, the stripes are merged on the client.

[0093] Different asynchronous threads in the client perform unified scheduling and management on the connected storage nodes. Asynchronous threads include submission threads, migration threads, and deletion threads. The storage node includes a cache layer and a hard disk drive (HDD). The cache layer includes memory and a solid-state drive (SSD), which serve as the first-level cache and second-level cache of the storage node, respectively. The data migration module in the storage node is used to migrate the data on the second-level cache SSD to the hard disk drive (HDD) after writing the data to the second-level cache SSD; the data cleaning module is used to delete the original file data on the second-level cache solid-state drive (SSD) after the data file on the second-level cache SSD is migrated.

[0094] When all slices of the target data are written to the storage node's primary cache, the client stores the corresponding target data information in the commit queue commmitQueue according to the storage node's first response; when all slices of a target data are written from the primary cache to the secondary cache in the storage node, the client stores the corresponding target data information in the migration queue migrateQueue according to the storage node's second response; when the storage node migrates the secondary cache SSD to the HDD, it returns confirmation information to the client, and the client stores the target data information in the delete queue deleteQueue. In each queue, write requests, migration requests, and delete requests are sent to the storage node through the corresponding asynchronous thread.

[0095] In some of the embodiments, after the client receives the first response from the storage node, the target data information corresponding to the slice data in the completed state is stored in the submission queue; in the submission queue, the second write request corresponding to the target data is sent to the storage node through the submission thread.

[0096] Specifically, after the client receives the first responses returned by multiple storage nodes, if all slice data are in the completion status finish, the target data information (for example, target data ID) corresponding to the slice data is stored in the client's submission queue, and the submission thread periodically consumes a target data information record from the submission queue, and issues a second write request to write the data slice of the target data from the storage node's first-level cache to the second-level cache SSD.

[0097] Figure 6 FIG. 4 is a schematic diagram of the process of issuing and executing the second write request in this embodiment. Figure 6 As shown, after the storage node sends the first response to the client, the client performs status judgment and stores the target data information corresponding to the slice data in the finished state finish in the submission queue; in the submission queue, the second write request corresponding to the target data is sent to the storage node through the submission thread. In the storage node, according to the target data information, the corresponding slice object is determined in the completion set finish_piece_map, and according to the load balancing strategy, the write path of the slice data is determined, that is, the specific secondary cache SSD and HDD to be written, and the path information ssdPath and hddPath in the above data information are updated accordingly. After the slice data is successfully written to the ssdPath path of the secondary cache, the slice object is reinitialized and placed in the initial set idle_piece_list. At the same time, the target data information corresponding to the slice data is added to the data migration module migrate_piece_list, and the second response of successful writing is returned to the client.

[0098] In this embodiment, after the client receives the first response from the storage node, the target data information corresponding to the slice data in the completed state is stored in the submission queue; in the submission queue, the second write request corresponding to the target data is sent to the storage node through the submission thread, and the client implements unified scheduling management of each storage node, thereby ensuring the data integrity and synchronization of the storage node's first-level cache writing to the second-level cache.

[0099] In some of the embodiments, it also includes:

[0100] When the client receives the second response of the storage node to the second write request, the target data information corresponding to the successfully written slice data is stored in the migration queue; in the migration queue, the migration request corresponding to the target data is sent to the storage node through the migration data thread.

[0101] Specifically, after the client receives the second response that the secondary cache has been successfully written from all storage nodes, the target data information (for example, the target data ID) corresponding to the current slice data is added to the migration queue migrateQueue, and a target data information record is consumed from the migration queue migrateQueue through the migration thread, and a migration request is issued to the storage node, requesting that the slice data of the target data be migrated from the secondary cache SSD to the HDD disk.

[0102] In some of the embodiments, it also includes:

[0103] When the storage node receives a migration request for target data, it writes the complete slice data of each target data in the secondary cache to the HDD of the storage node.

[0104] Specifically, Figure 7 FIG. 1 is a schematic diagram of the process of issuing and executing a migration request in this embodiment. Figure 7 As shown, after the client receives the second response of successful writing to the secondary cache from all storage nodes, it adds the target data information (e.g., target data ID) corresponding to the current slice data to the migration queue migrateQueue, consumes a target data information record from the migration queue migrateQueue through the migration thread, and issues a migration request to the storage node, requesting to migrate the slice data of the target data from the secondary cache SSD to the HDD disk. The storage node receives the migration request issued by the client, finds the SSD path and HDD path of the slice data according to the target data information from the data migration module migrate_piece_list, and then migrates the slice data of the SSD path to the HDD path, and stores the slice data information in the data cleaning module delete_piece_list, and feeds back the migration result to the client as the third response.

[0105] In this embodiment, the client implements unified scheduling management of each storage node, thereby ensuring the data integrity and synchronization of the storage node secondary cache migrated to the HDD disk.

[0106] In some of the embodiments, it also includes:

[0107] When the client receives the third response from the storage node to the migration request, the target data information corresponding to the successfully written slice data is stored in the deletion queue; in the deletion queue, the deletion request corresponding to the target data is sent to the secondary cache through the deletion thread.

[0108] Specifically, Figure 8 FIG. 4 is a schematic diagram of the process of sending and executing a delete request in this embodiment. Figure 8 As shown, after the client receives the third response of the data slice migration result of all storage nodes, it stores the target data information corresponding to the slice data in the deletion queue deleteQueue, consumes the target data information record from the deletion queue deleteQueue through the deletion thread, and issues a deletion request for the slice data of the target data on the secondary cache SSD. The storage node receives the deletion request issued by the client, finds the slice data corresponding to the target data from the data cleaning module delete_piece_list according to the target data information, deletes the slice data corresponding to the target data under ssdPath according to the ssdPath path information of the slice data, deletes the slice data from the data cleaning module delete_piece_list at the same time, and sends the deletion result to the client. After the client receives the successful response of all storage nodes deleting the slice data under the secondary cache SSD path, it deletes the corresponding target data information from the deletion queue deleteQueue.

[0109] The present embodiment is described and illustrated below through preferred embodiments.

[0110] Fig. 9 is a flowchart of the multi-level cache method in this embodiment, which is applied in the above embodiments. Figure 5 Cloud storage system. Fig. 9 As shown, the method comprises the following steps:

[0111] Step S910: In the client, the target data to be written is divided into a number of slice data, and each slice data is then divided into the same number of data blocks; the data blocks at the corresponding position of each slice data form a stripe.

[0112] Step S920, merging the stripes that meet the preset merging rule in the client; the preset merging rule is that the stripes belong to the same target data and the memory addresses of the stripes are continuous.

[0113] Step S930, when the storage node receives the first write request for the target data, it merges the stripes in the client and writes them into the first-level cache; according to the status of the slice data in the storage node, it adopts the corresponding strategy to update the status; and returns the first response to the first write request to the client.

[0114] Step S940, after the client receives the first response from the storage node, the target data information corresponding to the slice data in the completed state is stored in the submission queue; in the submission queue, the second write request corresponding to the target data is sent to the storage node through the submission thread.

[0115] Step S950, when the client receives the second response of the storage node to the second write request, the target data information corresponding to the successfully written slice data is stored in the migration queue; in the migration queue, the migration request corresponding to the target data is sent to the storage node through the migration data thread.

[0116] Step S960, when the storage node receives a migration request for target data, it migrates the complete slice data of each target data in the secondary cache to the HDD of the storage node, and returns a third response of the migration result to the client.

[0117] Step S970, when the client receives the third response from the storage node to the migration request, the target data information corresponding to the successfully written slice data is stored in the deletion queue; in the deletion queue, the deletion request corresponding to the target data is sent to the secondary cache through the deletion thread.

[0118] Step S980: Delete the slice data corresponding to the target data in the secondary cache in the storage node, and send the deletion result to the client.

[0119] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0120] In this embodiment, a multi-level cache device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below may be a combination of software and / or hardware that implements predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0121] Fig.10is a structural block diagram of a multi-level cache device of this embodiment, which is applied to a cloud storage system. The cloud storage system includes a client and several storage nodes. The storage nodes include a primary cache and a secondary cache. Fig.10 As shown, the device comprises:

[0122] The data slicing module 10 is used to divide the target data in the client into a plurality of slice data; the slice data is stored in the form of stripes;

[0123] A first cache module 20, configured to merge the stripes in the client and write them into the first-level cache when the storage node receives a first write request for target data;

[0124] The second cache module 30 is used to write the complete slice data of each target data in the first-level cache into the second-level cache when the storage node receives a second write request for the target data.

[0125] Through the device provided in this embodiment, the merged stripes in the client can be written into the first-level cache of the storage node, and written into the second-level cache after the first-level cache receives the complete slice data of each target data, thereby reducing the network interaction and small IO operations between the client and the storage node, improving the reading and writing efficiency, and solving the problem of a large number of small-scale IO operations affecting the overall performance of the system.

[0126] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0127] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0128] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0129] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0130] In addition, in combination with the multi-level cache method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the multi-level cache method. The storage medium stores a computer program, and when the computer program is executed by a processor, any multi-level cache method in the above embodiments is implemented.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0132] It should be understood that the specific embodiments described here are only used to explain this application, rather than to limit it. According to the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0133] Obviously, the drawings are only some examples or embodiments of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complex and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. The term "embodiment" in this application refers to the specific features, structures or characteristics described in conjunction with the embodiment that may be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clear or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments in the absence of conflict.

[0134] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A multi-level cache method, characterized in that: Applied to a cloud storage system, the cloud storage system includes a client and several storage nodes, the storage nodes include a primary cache and a secondary cache; the method includes: The target data in the client is divided into a number of slice data; the slice data is stored in the form of strips; the target data is picture data or small files in the video cloud storage; When the storage node receives the first write request for the target data, the storage node merges the stripes in the client and writes them into the first-level cache; When the storage node receives a second write request for the target data, writing the complete slice data of each target data in the first-level cache into the second-level cache; According to the response of the storage node to the write request of the client, the storage node is uniformly scheduled and managed through the corresponding asynchronous thread in the client.

2. The multi-level cache method according to claim 1, characterized in that: Also includes: Merging the stripes satisfying a preset merging rule in the client; The preset merging rule is that the stripes belong to the same target data and the memory addresses of the stripes are continuous.

3. The multi-level cache method according to claim 1, characterized in that: The step of merging the stripes in the client and writing them into the first-level cache further includes: According to the status of the slice data in the storage node, a corresponding strategy is adopted to update the status; and a first response to the first write request is returned to the client.

4. The multi-level cache method according to claim 3, characterized in that: Also includes: When the client receives the first response from the storage node, the target data information corresponding to the slice data in the completed state is stored in the submission queue; In the submission queue, a second write request corresponding to the target data is sent to the storage node through a submission thread.

5. The multi-level cache method according to claim 1, characterized in that: Also includes: When the client receives the second response of the storage node to the second write request, the client stores the target data information corresponding to the successfully written slice data in the migration queue; In the migration queue, a migration request corresponding to the target data is sent to the storage node through a migration data thread.

6. The multi-level cache method according to claim 1, characterized in that: Also includes: When the storage node receives a migration request for the target data, each complete slice data of the target data in the secondary cache is migrated to the HDD of the storage node.

7. The multi-level cache method according to claim 6, characterized in that: Also includes: When the client receives the third response of the storage node to the migration request, the target data information corresponding to the successfully written slice data is stored in the deletion queue; In the deletion queue, a deletion request corresponding to the target data is sent to the secondary cache through a deletion thread.

8. A multi-level cache device, characterized in that: Applied to a cloud storage system, the cloud storage system includes a client and several storage nodes, the storage nodes include a primary cache and a secondary cache; the device includes: A data slicing module is used to divide the target data in the client into a number of slice data; the slice data is stored in the form of strips; the target data is picture data or small files in the video cloud storage; A first cache module, configured to merge the stripes in the client and write them into the first-level cache when the storage node receives a first write request for the target data; A second cache module, configured to write complete slice data of each target data in the first-level cache into the second-level cache when the storage node receives a second write request for the target data; According to the response of the storage node to the write request of the client, the storage node is uniformly scheduled and managed through the corresponding asynchronous thread in the client.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the multi-level cache method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-level cache method according to any one of claims 1 to 7 are implemented.

Images