A Ceph Distributed Object Storage Directory Retrieval Method and Device

Abstract

The present invention discloses a method and device for retrieving directories in a Ceph distributed object storage, including: obtaining the access times of each directory structure, and respectively dividing the directory structures into a hot linked list, a warm linked list, and a cold linked list according to the access times; when receiving a retrieval request for a target directory structure, obtaining the retrieval type of the retrieval request, where the retrieval type includes one or several of a write request, a read request, a delete request, a copy request, and a move request; and sequentially retrieving the target directory structure in the hot linked list, the warm linked list, and the cold linked list according to the retrieval type. During the above retrieval process, each directory structure is pre-divided into a hot linked list, a warm linked list, and a cold linked list according to the access times, and the target directory structure is sequentially retrieved in the hot linked list, the warm linked list, and the cold linked list, eliminating the need to retrieve directories by enumeration, thus improving the retrieval efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage, and in particular, to a method and device for retrieving directories in a Ceph distributed object storage system. Background Art

[0002] Ceph is a distributed storage system with high performance, high availability, high scalability, and rich features, supporting file storage, object storage, and block storage. In object storage, the RGW component (rados gateway) provided by Ceph can support docking with the S3 protocol and the Switch protocol. Clients can perform functions such as uploading / downloading / deleting / retrieving objects (files) through the protocol. At the same time, Ceph object storage supports mounting buckets using nfs-ganesha and accessing the content of object storage in a way similar to file storage, greatly improving the convenience of accessing object storage.

[0003] Due to the characteristics of object storage itself, it does not have a file directory tree structure like a file system. Therefore, the retrieval of the object system can only be performed by prefix. All objects that meet the prefix are enumerated in the osd, and then the specific content of the request is used to extract all the enumerated results. However, in many cases, users only want to enumerate the subdirectories or file objects under a certain directory. For the enumeration of file objects, the current mechanism can meet most retrieval requests because object storage retrieval of objects is inherently to enumerate these objects, so as long as the number of enumerated objects is returned to the upper layer. But for the enumeration of directories, especially in the case where there are many nested directories, the current retrieval mechanism is very inefficient and can only extract the bottom-layer objects of these nested directories after enumeration. Summary of the Invention

[0004] In view of this, the present invention provides a method and device for retrieving directories in a Ceph distributed object storage system, which is used to solve the problem that the current retrieval mechanism is very inefficient for the enumeration of directories, especially in the case where there are many nested directories, and can only extract the bottom-layer objects of these nested directories after enumeration. The specific solutions are as follows:

[0005] A method for retrieving directories in a Ceph distributed object storage system includes:

[0006] Obtaining the access times of each directory structure, and respectively dividing each directory structure into a hot linked list, a warm linked list, and a cold linked list according to the access times;

[0007] When receiving a retrieval request for a target directory structure, obtaining the retrieval type of the retrieval request, where the retrieval type includes one or more of a write request, a read request, a delete request, a copy request, and a move request;

[0008] Retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type.

[0009] For the above method, optionally, divide each directory structure into the hot linked list, the warm linked list, and the cold linked list according to the access times, including:

[0010] Judge whether the current access times is greater than the preset access times threshold. If so, divide the directory structure corresponding to the current access times into the hot linked list. If not, divide the directory structure corresponding to the current access times into the cold linked list;

[0011] In the case where the length of the linked list in the hot linked list is greater than the maximum length of the hot linked list, divide the part in the hot linked list that is greater than the maximum length into the warm linked list;

[0012] In the case where it is detected that any target structure directory is divided into the warm linked list, start timing. If it is still in the warm linked list after a preset duration, add it to the cold linked list.

[0013] For the above method, optionally, further include:

[0014] If the current directory structure in the hot linked list is accessed when it is greater than the hot linked list window value, move the current directory structure back to the head of the hot linked list.

[0015] For the above method, optionally, in the case where the retrieval type is a write request and a copy request, retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type, including:

[0016] Judge whether the target directory structure is stored in the hot linked list. If so, obtain the actual number of layers and the position of the target directory structure, and make the number of direct subdirectories or direct objects recorded by the target directory structure +1 or remain unchanged according to the actual number of layers and the position;

[0017] If not, judge whether the target directory structure is stored in the warm linked list;

[0018] If the target directory structure is not stored in the warm linked list, check whether the target directory structure exists in the cold linked list. If the target directory structure exists in the cold linked list, update the actual number of layers and the position of the target directory structure, or, if the target directory structure does not exist in the cold linked list, add the target directory structure to the cold linked list.

[0019] The above method, optionally, in the case where the retrieval type is a read request, sequentially retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list according to the retrieval type, including:

[0020] Determine whether the target directory structure is stored in the hot linked list. If so, directly read the direct directory tree and the number of direct objects in the target directory structure;

[0021] If not, determine whether the target directory structure is stored in the warm linked list;

[0022] If the target directory structure is not stored in the warm linked list, check whether the target directory structure exists in the cold linked list. If the target directory structure exists in the cold linked list, read the target directory structure in the osd, and then increment the access count of the target directory structure by one. If the target directory structure does not exist in the cold linked list, read the target directory structure in the osd and asynchronously write it to the cold linked list.

[0023] The above method, optionally, further includes:

[0024] If the target directory structure exists in the warm linked list but the existence duration exceeds the preset duration, delete the target directory structure from the warm linked list and add it to the cold linked list, and reset the access count;

[0025] If the target directory structure exists in the warm linked list but the existence duration does not exceed the preset duration, update the record information in the warm linked list, delete the target directory structure from the warm linked list and add it to the hot linked list.

[0026] The above method, optionally, in the case where the retrieval type is a delete request, sequentially retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list according to the retrieval type, including:

[0027] Identify the deletion type included in the delete request, where the deletion type is an object or a directory;

[0028] In the case where the deletion type is a directory, obtain the parent directory of the directory in the target directory structure;

[0029] Sequentially determine whether the directory exists in the hot linked list, the warm linked list, and the cold linked list. If the directory exists in any of the linked lists, delete the directory;

[0030] Sequentially determine whether the parent directory exists in the hot linked list, the warm linked list, and the cold linked list. If the parent directory exists in any of the linked lists, delete the parent directory.

[0031] A Ceph distributed object storage directory retrieval device, comprising:

[0032] An acquisition and division module, configured to acquire the access times of each directory structure, and divide each of the directory structures into a hot linked list, a warm linked list, and a cold linked list according to the access times;

[0033] An acquisition module, configured to, when receiving a retrieval request for a target directory structure, acquire the retrieval type of the retrieval request, where the retrieval type includes one or more of: a write request, a read request, a delete request, a copy request, and a move request;

[0034] A retrieval module, configured to sequentially retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list according to the retrieval type.

[0035] For the above-mentioned device, optionally, the acquisition and division module includes:

[0036] A first judgment unit, configured to judge whether the current access times is greater than a preset access times threshold. If so, divide the directory structure corresponding to the current access times into the hot linked list; if not, divide the directory structure corresponding to the current access times into the cold linked list;

[0037] A division unit, configured to, when the length of the linked list in the hot linked list is greater than the maximum length of the hot linked list, divide the part greater than the maximum length in the hot linked list into the warm linked list;

[0038] A timing and addition unit, configured to start timing when detecting that any target structure directory is divided into the warm linked list, and add it to the cold linked list if it is still in the warm linked list after a preset duration.

[0039] For the above-mentioned device, optionally, when the retrieval type is a write request and a copy request, the retrieval module includes:

[0040] A second judgment unit, configured to judge whether the target directory structure is stored in the hot linked list. If so, acquire the actual number of layers and the position of the target directory structure, and make the number of direct sub-directories or direct objects recorded by the target directory structure +1 or remain unchanged according to the actual number of layers and the position;

[0041] A third judgment unit, configured to, if not, judge whether the target directory structure is stored in the warm linked list;

[0042] A search unit is configured to search in the cold chain table for the target directory structure if it is not stored in the warm chain table. If the target directory structure exists in the cold chain table, update the actual level and location of the target directory structure. Or, if the target directory structure does not exist in the cold chain table, add the target directory structure to the cold chain table.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] The present invention discloses a Ceph distributed object storage directory retrieval method and apparatus, including: obtaining the access times of each directory structure, and respectively dividing each directory structure into a hot chain table, a warm chain table, and a cold chain table according to the access times; when receiving a retrieval request for a target directory structure, obtaining the retrieval type of the retrieval request, where the retrieval type includes one or more of a write request, a read request, a delete request, a copy request, and a move request; and sequentially retrieving the target directory structure in the hot chain table, the warm chain table, and the cold chain table. In the above retrieval process, each directory structure is pre-divided into a hot chain table, a warm chain table, and a cold chain table according to the access times, and the target directory structure is sequentially retrieved in the hot chain table, the warm chain table, and the cold chain table, and there is no need to retrieve the directory by enumeration, which improves the retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0046] Figure 1 It is a flowchart of a Ceph distributed object retrieval method disclosed in the prior art;

[0047] Figure 2 It is a flowchart of a Ceph distributed object storage directory retrieval method disclosed in an embodiment of the present application;

[0048] Figure 3 It is a schematic diagram of a three-chain table structure disclosed in an embodiment of the present application;

[0049] Figure 4 It is another flowchart of a Ceph distributed object storage directory retrieval method disclosed in an embodiment of the present application;

[0050] Figure 5Another flowchart of a Ceph distributed object storage directory retrieval method disclosed in an embodiment of the present application;

[0051] Figure 6 Another flowchart of a Ceph distributed object storage directory retrieval method disclosed in an embodiment of the present application;

[0052] Figure 7 Another flowchart of a Ceph distributed object storage directory retrieval method disclosed in an embodiment of the present application;

[0053] Figure 8 Another flowchart of a Ceph distributed object storage directory retrieval method disclosed in an embodiment of the present application;

[0054] Figure 9 Block diagram of a Ceph distributed object storage directory retrieval device structure disclosed in an embodiment of the present application. Detailed implementation manners

[0055] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0056] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0057] The present invention discloses a storage directory retrieval method and device, which are applied to the process of directory retrieval based on the Ceph system. Ceph is a distributed storage system with high performance, high availability, high scalability and rich features, supporting file storage, object storage and block storage. Among them, in object storage, the RGW component (rados gateway) provided by Ceph can support docking the S3 protocol and the Switch protocol, and the client can perform functions such as uploading / downloading / deleting / retrieving objects (files) through the protocol. At the same time, ceph object storage supports mounting buckets using nfs-ganesha and accessing the content of object storage in a manner similar to file storage, greatly improving the convenience of accessing object storage.

[0058] Among them, whether directly using common clients such as s3cmd or mounting using protocols such as ganesha, retrieving objects is a very common function in object storage. Retrieving objects in object storage belongs to the readdir request. For example, Figure 1 is shown as follows:

[0059] a. After the client (S3cm) initiates a readdir request, it first reaches the RGW module at the Ceph end, that is, the object storage gateway module RGW. RGW determines whether it has finished reading (!truncate) or is executing for the first time.

[0060] b. The RGW module determines that it is the first execution or has finished reading. The readdir request reaches ceph-radosgw, creates a corresponding handler according to the type of the client, and creates a ListBucket object.

[0061] c. The ListBucket object enters the list_object process, performs object reading operations, sends the request down to the rados layer, and finally goes down to the osd for reading.

[0062] d. The osd returns the retrieved objects. RGW judges and processes these results, and also adds a flag indicating whether the retrieval is completed to the results. Briefly speaking, if the readdir request is to retrieve the final file, these objects are directly returned to the upper layer, and after verification, they are returned to the client; if the readdir request is for the files / directories under a certain directory, the results will be extracted, and the qualified results after extraction are returned to the upper layer until the client.

[0063] e. The client judges whether there is a completion flag in the results. If it has been completed, the client will display the results; if not, it will send another readdir request, bringing the marker flag, that is, the last result of the previous retrieval, and initiate a new retrieval process until the retrieval is completed.

[0064] Due to the characteristics of object storage itself, it does not have a file directory tree structure like a file system. Therefore, the retrieval of the object system can only be through a prefix. All objects that meet the prefix are enumerated in the OSD, and then the specific content of the request is used to extract all the enumerated results. However, in many cases, users only want to enumerate the subdirectories or file objects under a certain directory level. For the enumeration of file objects, the current mechanism can meet most retrieval requests because the object storage retrieves objects by enumerating these objects. So, as long as the number of enumerated objects is returned to the upper layer. But for the enumeration of directories, especially in the case where there are many nested directories, the current retrieval mechanism is very inefficient. It can only enumerate the bottom-layer objects of these nested directories and then extract them. For example: a directory a has b subdirectories, each of the b subdirectories has c subdirectories, and each of the c subdirectories has d file objects. In such a tree-structured directory, if you now want to retrieve the direct subdirectories under directory a, in object storage, currently, you can only enumerate all the leaf nodes belonging to directory a in this tree, and based on the paths of all leaf nodes, extract which direct subdirectories are under directory a. So, it is relatively inefficient. Therefore, if the directory tree structure is larger and more complex, this retrieval process will be very slow. Combining the above example, it can be known that the deeper and wider the directory tree is, the longer the time for retrieving the upper-level directories will be. Because of the containment relationship, the bottom-layer leaf nodes all belong to the middle and upper-level directories, so the leaf nodes are very large. And the retrieval time for the bottom-layer directories is still acceptable because the number of leaf nodes belonging to the lower-level directories is much smaller than the number of leaf nodes belonging to the middle and upper-level directories. Therefore, optimizing the retrieval for the middle and upper-level directories is a very necessary action.

[0065] Therefore, to solve the above problems, the present invention provides a method for retrieving directories in a Ceph distributed object storage. The execution process of the method is as Figure 2 shown, including the steps:

[0066] S101. Obtain the access times of each directory structure, and divide each directory structure into a hot linked list, a warm linked list, and a cold linked list according to the access times;

[0067] In the embodiments of the present invention, the access times of each directory structure are obtained in advance, where the directory structure contains at least one directory and / or object. In order to effectively record frequently used directory structures and quickly read them when the client retrieves directories, in the embodiments of the present invention, a three-linked list form is adopted to represent different access heat degrees, and the directory structures are cached and recorded. Each linked list is protected by a read-write lock to ensure data accuracy under concurrent conditions. Except for the search linked list, the process of writing to the linked list is an asynchronous operation. The three linked lists are: hot_list: hot linked list, warm_list: warm linked list, and cold_list: cold linked list. Among them, the schematic diagram of the three linked lists is as shown in Figure 3 shown, and cold_list includes:

[0068]

[0069] Among them, max_lru in cold_list: 6

[0070] warm_list includes:

[0071]

[0072] Among them, max_len: 6, max_time: T10

[0073] hot_list includes:

[0074]

[0075] Among them, max_len: 6, window: 3

[0076] hot_list represents the directory structure with the highest access heat degree, and the joining criterion is the directory structure with the access times >= N (preset access times threshold).

[0077] Characteristics:

[0078] hot_list resides in memory permanently and has a maximum length. When the length of the linked list is greater than the maximum length, the directory structure at the tail of the linked list is eliminated and added to the head of warm_list.

[0079] Operations:

[0080] The directory structures added to the hot_list are all added to the head of the hot_list first. As other directory structures are added, they are gradually pushed backward until they are pushed to the tail of the hot_list and eliminated. For example, Name: CC. To reduce frequent operations on the hot_list, a hot list window value hot_list_window is set in the hot_list. By default, it can be set to half of the hot_list length, or adjusted to other length values (less than the list length). If a directory structure added to the hot_list is accessed again while it is in the hot_list, if it is within the hot_list_window at this time, no operation is required. If it is outside the hot_list_window, this directory needs to be readjusted to the head of the hot_list. For example, Name: HH.

[0081] The warm_list represents the directory structures eliminated from the hot_list. The addition criterion is the directory structures with fewer access times in the hot_list.

[0082] Features:

[0083] The warm_list does not reside in memory permanently. It has a maximum length and there is a timer on it, which starts timing when a directory structure is added to the warm_list.

[0084] Operations:

[0085] The directory structures added to the warm_list are also added to the head of the warm_list first, and the timing is started. As other directory structures are added, they are pushed backward until they are pushed to the tail of the warm_list and eliminated and deleted. However, there is no warm_list_window on the warm_list. In addition to elimination at the tail, when the time of a directory structure exceeds a preset duration (checked during access), where the preset duration can be set based on experience or specific circumstances, and is not specifically limited in the embodiments of the present invention. If the directory structure is still in the warm_list, it is eliminated and deleted and added to the cold_list. For example, Name: KK and Name: OO, and the heat value is reset to N - 1 (N is the critical value for adding to the hot_list). When a directory structure in the warm_list is accessed, it is deleted from the warm_list and at the same time re-added to the head of the hot_list, following the operations of the hot_list.

[0086] The cold_list represents directory structures with low access heat. The addition criterion is directory structures with an access count <= N.

[0087] Features:

[0088] The cold_list resides in memory permanently and has a maximum length. When the length of the linked list exceeds the maximum length, the directory structure at the tail of the linked list is eliminated and deleted.

[0089] Operations:

[0090] The directory structures added to the cold_list are also first added to the head of the cold_list. As other directory structures are added, they are pushed backward until they reach the tail of the cold_list and are eliminated and deleted. At the same time, a counter, that is, the access count, is started for the directory structures added to the cold_list. It is 1 when initially added, and it is incremented by 1 every time it is accessed. When it reaches the set value N, it is deleted from the cold_list and added to the head of the hot_list, Name: WW.

[0091] Furthermore, regardless of which linked list in the three-linked list it is added to, the directory structure is a data structure that records important mapping information. While a directory structure represents a certain level of directory itself, it also contains the number of direct subdirectories and direct object markers under its own directory, and records the names of direct subdirectories under the directory through a container. Under a newly created directory, the number of direct subdirectories is 0 and the number of objects is 0, and the direct subdirectory container is empty. Then, according to the level where the directory is located, the number of direct subdirectories and object markers are modified. If the number of direct subdirectories is not 0, the direct subdirectory path is added to the container of this directory and added to the three-linked list. When there are new writes later, if this directory is involved, the three-linked list will be traversed to search. After finding it, its data will be updated according to the actual situation.

[0092] S102. In the case of receiving a retrieval request for a target directory structure, obtain the retrieval type of the retrieval request, where the retrieval type includes one or more of: write request, read request, delete request, copy request, and move request;

[0093] In the embodiments of the present invention, the target directory structure may be a directory or an object, and the retrieval type can be distinguished according to different identifiers. A corresponding retrieval type is assigned to each identifier. In the embodiments of the present invention, the identifiers for each retrieval type are not limited. In the case of receiving a retrieval request for a target directory structure, obtain the identifier included in the retrieval request, and determine the retrieval type based on the identifier, where the retrieval type includes one or more of: write request, read request, delete request, copy request, and move request.

[0094] S103. Retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type.

[0095] In the embodiments of the present invention, in the case where the retrieval request is a write request, the specific execution process is asFigure 4 As shown, when writing to the target directory structure, the absolute path of the write request is split in an asynchronous thread. For example, the object to be written is: / home / dir1 / dir2 / dir3 / file1

[0096] Then the following are split out:

[0097] / home / dir1 / dir2 / dir3 /

[0098] / home / dir1 / dir2 /

[0099] / home / dir1 /

[0100] / home /

[0101] Then the above four parent directories split out are all independent directory structures. After splitting out each layer of directories (taking one of the directories as an example here), first check if the target directory structure exists in the hot_list:

[0102] 1-1. If it exists, update sub_dir, update sub_obj, and update Container. The specific update process is as follows: Obtain the target directory structure in the hot_list. According to the actual number of layers and position of the target directory structure, increment or keep unchanged the number of direct subdirectories recorded in it. If incremented, add its direct subdirectories to the container, and the number of direct objects also depends on the actual situation. If an object is written under this layer of directory, the number of objects is incremented, otherwise it remains unchanged. If at this time the directory structure is within the hot_list_window of the hot_list, the process ends after the update. If it is outside the hot_list_window, move the target directory structure to the head of the hot_list linked list.

[0103] 1-2. If it does not exist, then check the warm_list:

[0104] 1-2-1. If it exists and is not out of date, read and update the information recorded in it, update sub_dir, update sub_obj, and update Container, and at the same time delete it from the warm_list and re-add it to the head of the hot_list;

[0105] 1-2-2. If it exists but has timed out, delete it from the warm_list and add it to the head of the cold_list, and reset the access popularity Num to N-1 (N is the critical value for adding to the hot_list), update sub_dir, update sub_obj, and update Container. For example: Num = 4

[0106] 1-2-3. If the warm_list does not exist either, check if the target directory structure exists in the cold_list:

[0107] 1-2-3-1. If it exists, read the target directory structure and update the data according to the actual situation, update sub_dir, update sub_obj, and update Container, and then check if the access popularity num + 1 reaches N. If it does not reach N, increment the directory structure access popularity by 1 and end the process. If it has reached N, delete this target directory structure from the cold_list and add it to the hot_list.

[0108] 1-2-3-2. If it does not exist, directly create a directory structure object, update the data, add it to the cold_list, increment the access popularity Num by 1, and end the over process.

[0109] In the case where the retrieval request is a read request, the specific execution process is as Figure 5 shown. When the readdir request traverses the subdirectories / objects under a directory, it will first check if the target directory structure exists in the hot_list:

[0110] 2-1. If it exists, access the number of direct subdirectories and direct objects under this target directory structure. If the number of direct objects sub_obj is not 0, continue the original process of traversing objects in the osd to obtain the specific object metadata, and for subdirectories, directly read the direct subdirectories recorded in the container; if the number of direct subdirectories is 0, there is no need to go through the osd lookup process at all, and directly read the content recorded in the container container to obtain the direct subdirectories. At the same time, if this target directory structure is outside the hot_list_window of the hot_list at this time, move this target directory structure to the head of the hot_list. Other situations can end the process.

[0111] 2-2. If it does not exist, check if the warm_list contains the target directory structure:

[0112] 2-2-1. If it exists and is not out of date, the reading method is the same as in 2-1. After reading the subdirectories, return to the upper layer. And delete this target directory structure from the warm_list and add it back to the head of the hot_list.

[0113] 2-2-2. If it exists and is out of date, but this situation does not affect obtaining the desired result. So after obtaining the result and returning to the upper layer, delete this directory structure from the warm_list and add it to the head of the cold_list, and reset the access popularity Num to N - 1. For example, Num = 4.

[0114] 2-2-3. If it does not exist in the warm_list either, follow the original process to directly read all results from the OSD. And search in the cold_list:

[0115] 2-2-3-1. If it exists, and if the object and object metadata are successfully read from the OSD, then determine whether the access heat Num + 1 reaches N. If it reaches, delete it from the cold_list and add it to the hot_list. If it does not reach, directly increment the access heat Num by 1.

[0116] 2-2-3-2. If it does not exist, and if the reading from the OSD is successful, write the read result to the cold_list through the asynchronous writing process, and increment the access heat Num by 1.

[0117] In the case where the retrieval request is a deletion request, what can be deleted is an object or a directory. The specific execution process for deleting an object is as Figure 6 shown, deleting an object (which is actually the same as the write request):

[0118] 3-a-1. Only extract the parent directory where the object to be deleted is located, and search in the hot_list:

[0119] 3-a-1-1. If it exists, subtract 1 from the number of direct objects sub_obj under the recorded directory structure, and keep the rest unchanged. The rest follows the characteristics and principles of the hot_list (the same rules as writing an object).

[0120] 3-a-1-2. If it does not exist, search in the warm_list:

[0121] 3-a-1-2-1. If it exists in the warm_list, whether it times out or not, subtract 1 from the number of direct objects sub_obj under the recorded directory structure, and the rest follows the characteristics and rules of the warm_list (the same rules as writing an object).

[0122] 3-a-1-2-2. If it does not exist in the warm_list, search in the cold_list:

[0123] 3-a-1-2-2-1. If it exists in the cold_list and the access heat Num++, subtract 1 from the number of direct objects sub_obj under the parsed directory of the recorded one, and the rest follows the characteristics and rules of the cold_list (the same rules as writing an object).

[0124] 3-a-1-2-2-2. If it does not exist in the cold_list, to ensure consistency, do not perform other operations.

[0125] The deletion directory includes: deleting the directory itself and its parent directory. The process of deleting the directory itself is as follows Figure 7 As shown, from the perspective of the directory to be deleted for deleting the entire directory, check if it exists in the hot_list?

[0126] 3-b-1-a1. If it exists, directly delete it from the hot_list.

[0127] 3-b-1-a2. If it doesn't exist, search the warm_list:

[0128] 3-b-1-a2-1. If it exists, directly delete it from the warm_list.

[0129] 3-b-1-a2-2. If it doesn't exist, search the cold_list:

[0130] 3-b-1-a2-2-1. If it exists, directly delete it from the cold_list.

[0131] 3-b-1-a2-2-2. If it doesn't exist, no other operations are required and the process ends (over).

[0132] The process of deleting the parent directory of the directory is as follows Figure 8 As shown, from the perspective of the parent directory of the directory to be deleted for deleting the entire directory, check if it exists in the hot_list?

[0133] 3-b-1-b1. If it exists in the hot_list, read the information, subtract 1 from the direct directory count sub_obj, and delete the corresponding sub-directory structure in the remove container container, and the rest remains unchanged. Others follow the characteristics and rules of the hot_list (basically the same as the write process).

[0134] 3-b-1-b2. If it doesn't exist in the hot_list, search the warm_list:

[0135] 3-b-1-b2-1. If it exists in the warm_list, whether it times out or not, read the information, subtract 1 from the direct directory count sub_obj, and delete the corresponding sub-directory structure in the remove container container. Others follow the characteristics and rules of the warm_list (basically the same as the write process).

[0136] 3-b-1-b2-2. If it doesn't exist in the warm_list, search the cold_list:

[0137] 3-b-1-b2-2-1. If the cold_list exists, access heat, Num++, then read and directly add the directory number sub_obj-1, and delete the corresponding subdirectory structure in the remove container. Others follow the cold_list characteristics and rules (basically the same as the write process).

[0138] 3-b-1-b2-2-2.cold_list does not exist. To ensure consistency, no other operations are performed.

[0139] Furthermore, when the search request is a copy request and a move request (asynchronous), the copy executes the write process and can be executed completely according to the write process. The move executes the delete-then-write process and can be executed according to the above delete+write operation.

[0140] The present invention discloses a Ceph distributed object storage directory retrieval method, including: obtaining the number of accesses to each directory structure, and dividing each directory structure into a hot link list, a warm link list, and a cold link list according to the number of accesses; in the case of receiving a retrieval request for a target directory structure, obtaining the retrieval type of the retrieval request, the retrieval type including: one or more of a write request, a read request, a delete request, a copy request, and a move request; and searching the target directory structure in the hot link list, the warm link list, and the cold link list in sequence according to the retrieval type. In the above retrieval process, each directory structure is divided into a hot link list, a warm link list, and a cold link list in advance according to the number of accesses, and the target directory structure is searched in the hot link list, the warm link list, and the cold link list in sequence, and it is no longer necessary to search the directory by enumeration, thereby improving the retrieval efficiency.

[0141] Based on the above method, for the readdir request of the middle and upper directories with a large number of nested directories, it is only necessary to query the container content to obtain the subdirectories, and there is no need to list the objects one by one from the osd and extract them again. The access frequency of the middle and upper directories is very frequent, so the three-linked list cache method is suitable for recording these highly popular directory structures. The access frequency of the bottom directory is lower than that of the middle and upper directories. It is at the bottom of the directory tree and covers a relatively small number of leaf nodes, so the query itself is also faster. At the same time, in the normal writing and deleting object operations, although the management and processing mechanism of the linked list is added, it is all processed through asynchronous processes. That is, as long as the complete directory name to be processed is placed in the queue of another thread, the other thread will check and process these linked lists by itself, which will not affect the original business. Overall, combined with the original search process, the search efficiency of the entire object storage will be greatly improved.

[0142] Based on the above Ceph distributed object storage directory retrieval method, an embodiment of the present invention also provides a Ceph distributed object storage directory retrieval device. The structural block diagram of the retrieval device is as shown in Figure 9 shown, and includes:

[0143] An acquisition and partitioning module 201, an acquisition module 202, and a retrieval module 203.

[0144] Among them,

[0145] The acquisition and partitioning module 201 is configured to acquire the access times of each directory structure, and partition each directory structure into a hot linked list, a warm linked list, and a cold linked list according to the access times;

[0146] The acquisition module 202 is configured to, when receiving a retrieval request for a target directory structure, acquire the retrieval type of the retrieval request, where the retrieval type includes one or more of a write request, a read request, a delete request, a copy request, and a move request;

[0147] The retrieval module 203 is configured to sequentially retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list according to the retrieval type.

[0148] The present invention discloses a Ceph distributed object storage directory retrieval device, including: acquiring the access times of each directory structure, partitioning each directory structure into a hot linked list, a warm linked list, and a cold linked list according to the access times; when receiving a retrieval request for a target directory structure, acquiring the retrieval type of the retrieval request, where the retrieval type includes one or more of a write request, a read request, a delete request, a copy request, and a move request; sequentially retrieving the target directory structure in the hot linked list, the warm linked list, and the cold linked list according to the retrieval type. In the above retrieval process, each directory structure is pre-partitioned into a hot linked list, a warm linked list, and a cold linked list according to the access times, and the target directory structure is sequentially retrieved in the hot linked list, the warm linked list, and the cold linked list, eliminating the need to retrieve the directory by enumeration, thereby improving the retrieval efficiency.

[0149] In an embodiment of the present invention, the acquisition and partitioning module 201 includes:

[0150] A first judgment unit 204, a partitioning unit 205, and a timing and adding unit 206.

[0151] Among them,

[0152] The first determination unit 204 is configured to determine whether the current access count is greater than a preset access count threshold. If so, the directory structure corresponding to the current access count is divided into the hot linked list; if not, the directory structure corresponding to the current access count is divided into the cold linked list.

[0153] The division unit 205 is configured to, when the length of the linked list in the hot linked list is greater than the maximum length of the hot linked list, divide the part of the hot linked list that is greater than the maximum length into the warm linked list.

[0154] The timing and adding unit 206 is configured to start timing when it is detected that any target structure directory is divided into the warm linked list. If it remains in the warm linked list after a preset duration, add it to the cold linked list.

[0155] In an embodiment of the present invention, when the retrieval type is a write request and a copy request, the retrieval module 202 includes:

[0156] A second determination unit 207, a third determination unit 208, and a search unit 209.

[0157] Wherein,

[0158] The second determination unit 207 is configured to determine whether the target directory structure is stored in the hot linked list. If so, obtain the actual number of layers and the position of the target directory structure, and increment or keep unchanged the number of direct sub - directories or direct objects recorded by the target directory structure according to the actual number of layers and the position.

[0159] The third determination unit 208 is configured to, if not, determine whether the target directory structure is stored in the warm linked list.

[0160] The search unit 209 is configured to, if the target directory structure is not stored in the warm linked list, search whether the target directory structure exists in the cold linked list. If the target directory structure exists in the cold linked list, update the actual number of layers and the position of the target directory structure; or, if the target directory structure does not exist in the cold linked list, add the target directory structure to the cold linked list.

[0161] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on its device - related embodiments. Since it is basically similar to the method embodiments, the description is relatively simple. For the related parts, refer to the partial description of the method embodiments.

[0162] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0163] For the convenience of description, when describing the above device, it is divided into various units according to functions for separate description. Of course, when implementing the present invention, the functions of each unit can be implemented in one or more software and / or hardware.

[0164] From the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes contributions to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0165] The above has introduced in detail a Ceph distributed object storage directory retrieval method and device provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for retrieving directories in a Ceph distributed object storage, characterized in that, it includes: Obtain the access times of each directory structure, and divide each directory structure into a hot linked list, a warm linked list, and a cold linked list respectively according to the access times; When receiving a retrieval request for a target directory structure, obtain the retrieval type of the retrieval request, where the retrieval type includes one or several of: write request, read request, delete request, copy request, and move request; Retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type; Among them, dividing each directory structure into a hot linked list, a warm linked list, and a cold linked list respectively according to the access times includes: Judge whether the current access times is greater than a preset access times threshold. If so, divide the directory structure corresponding to the current access times into the hot linked list. If not, divide the directory structure corresponding to the current access times into the cold linked list; When the length of the linked list in the hot linked list is greater than the maximum length of the hot linked list, divide the part in the hot linked list that is greater than the maximum length into the warm linked list; When it is detected that any target structure directory is divided into the warm linked list, start timing. If it is still in the warm linked list after a preset time period, delete it from the warm linked list and add it to the cold linked list.

2. The method according to claim 1, characterized in that, it further includes: When the current directory structure in the hot linked list is accessed when it is greater than the hot linked list window value, move the current directory structure back to the head of the hot linked list; where the hot linked list window value is a value less than the linked list length.

3. The method according to claim 1, characterized in that, When the retrieval type is a write request and a copy request, retrieving the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type includes: Judge whether the target directory structure is stored in the hot linked list. If so, obtain the actual number of layers and position of the target directory structure, and make the number of direct sub - directories or direct objects recorded by the target directory structure +1 or remain unchanged according to the actual number of layers and the position; If not, judge whether the target directory structure is stored in the warm linked list; If the target directory structure is not stored in the warm linked list, check whether the target directory structure exists in the cold linked list. If the target directory structure exists in the cold linked list, update the actual number of layers and position of the target directory structure, or, if the target directory structure does not exist in the cold linked list, add the target directory structure to the cold linked list.

4. The method according to claim 1, characterized in that, When the retrieval type is a read request, retrieving the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type includes: Determine whether the target directory structure is stored in the hot linked list. If so, directly read the direct directory tree and the number of direct objects in the target directory structure; If not, determine whether the target directory structure is stored in the warm linked list; If the target directory structure is not stored in the warm linked list, check whether the target directory structure exists in the cold linked list. If the target directory structure exists in the cold linked list, read the target directory structure in the osd, and then increment the access count of the target directory structure by one. If the target directory structure does not exist in the cold linked list, read the target directory structure in the osd and asynchronously write it to the cold linked list.

5. The method according to claim 3 or 4, wherein, further comprising: If the target directory structure exists in the warm linked list but the existence duration exceeds the preset duration, delete the target directory structure from the warm linked list and add it to the cold linked list, and reset the access count; If the target directory structure exists in the warm linked list but the existence duration does not exceed the preset duration, update the record information in the warm linked list, delete the target directory structure from the warm linked list and add it to the hot linked list.

6. The method according to claim 1, wherein, In the case where the retrieval type is a deletion request, retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type, including: Identify the deletion type included in the deletion request, wherein the deletion type is an object or a directory; In the case where the deletion type is a directory, obtain the parent directory of the directory in the target directory structure; Sequentially determine whether the directory exists in the hot linked list, the warm linked list, and the cold linked list. If the directory exists in any of the linked lists, delete the directory; Sequentially determine whether the parent directory exists in the hot linked list, the warm linked list, and the cold linked list. If the parent directory exists in any of the linked lists, delete the parent directory.

7. A Ceph distributed object storage directory retrieval device, wherein, comprising: An acquisition and partitioning module, configured to acquire the access counts of each directory structure, and partition each directory structure into a hot linked list, a warm linked list, and a cold linked list according to the access counts; An acquisition module, configured to acquire the retrieval type of the retrieval request when receiving a retrieval request for a target directory structure, wherein the retrieval type includes one or more of a write request, a read request, a deletion request, a copy request, and a move request; A retrieval module, configured to retrieve the target directory structure in the hot linked list, the warm linked list, and the cold linked list in sequence according to the retrieval type; wherein, the acquisition and partitioning module includes: A first judgment unit, configured to judge whether the current access count is greater than a preset access count threshold. If so, partition the directory structure corresponding to the current access count into the hot linked list. If not, partition the directory structure corresponding to the current access count into the cold linked list; A division unit, configured to divide the part of the hot linked list that is greater than the maximum length of the hot linked list into the warm linked list when the length of the linked list in the hot linked list is greater than the maximum length of the hot linked list; A timing and adding unit, configured to start timing when it is detected that any target structure directory is divided into the warm linked list, and if it is still in the warm linked list after a preset duration, delete it from the warm linked list and add it to the cold linked list.

8. The apparatus according to claim 7, wherein, when the retrieval type is a write request and a copy request, the retrieval module includes: A second judgment unit, configured to judge whether the target directory structure is stored in the hot linked list, and if so, obtain the actual number of layers and the position of the target directory structure, and according to the actual number of layers and the position, make the number of direct subdirectories or direct objects recorded by the target directory structure +1 or remain unchanged; A third judgment unit, configured to, if not, judge whether the target directory structure is stored in the warm linked list; A search unit, configured to, if the target directory structure is not stored in the warm linked list, search whether the target directory structure exists in the cold linked list, and if the target directory structure exists in the cold linked list, update the actual number of layers and the position of the target directory structure, or, if the target directory structure does not exist in the cold linked list, add the target directory structure to the cold linked list.

Images