A Ceph-based distributed object storage retrieval method and device

Abstract

The present invention discloses a Ceph-based distributed object storage retrieval method and device, which is applied to a Ceph system. The method receives a retrieval object request initiated by a client, determines s_max based on the retrieval object request, determines whether the last retrieval object adjacent to the current retrieval object is a file or a directory, determines max based on s_max if the object is a directory, reads the current retrieval object a preset number of times based on max, adjusts max to a preset number, and continues reading based on the preset number until all directories on the current layer are read; and adjusts max based on the number of files in the current retrieval object, reads the current retrieval object based on max until all directories on the current layer are read, and the number of files is positively correlated with max. The above process distinguishes the type of the current retrieval object, and adopts different max values ​​for flexible adjustment when the object is a file or a directory. This can improve retrieval efficiency when there are many retrieval objects.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a Ceph-based distributed object storage retrieval method and device. Background Art

[0002] Currently, in Ceph object storage, the number of objects read from the OSD in each readdir request sent by the client is fixed at the default value of max = 1000 - attempt, regardless of the number of objects in the directory layer. Therefore, this has two disadvantages:

[0003] If all objects under the directory at this level are files or most of them are files, when the number of files is small, the reading speed is relatively fast. However, when the number of files is very large, such as tens of thousands or even hundreds of thousands of files, only about 9,000 files can be read at a time. Multiple requests need to be sent between the client and the Ceph side to read all of them. In addition, the interaction between the RGW in the Ceph side and the OSD in the Rados layer needs to interact at least 9 times for each Readdir request. This frequent interaction is very time-consuming. In this case, the indicator 1,000 is too small. It is hoped that the max value and the attempt value should be as large as possible to reduce the number of interactions between modules.

[0004] If all or most of the objects in the current directory are directories, each read request will read max = 1000 - attempt objects at the bottom of the directory. Similarly, after a maximum of nine attempts, the directory at the current level is extracted from all the objects, and then the next readdir request is made to the upper level. When the number of directories in the current directory is small or moderate, only the current directory is needed, that is, only the object at the bottom of the current directory is read from the OSD, and the extra 999 - attempt objects are not needed. Therefore, a smaller max value is better and a larger attempt value is better. When the number of directories in the current directory is very large, max should not be too small, otherwise it will send too many requests to the OSD. It is better to read as many directories in the current directory as possible at once. Therefore, a larger max value and a larger attempt value are better.

[0005] The main problem with the above two points is that the max value is fixed and cannot be changed at any time according to the directory or file situation, resulting in the inability to make flexible adjustments. Therefore, when the number of objects is large, the retrieval efficiency is low. Summary of the Invention

[0006] In view of this, the present invention provides a Ceph-based distributed object storage retrieval method and device to solve the problem that the max value is fixed and cannot be changed at any time according to the directory or file situation in the existing Ceph-based distributed object retrieval process, resulting in inflexible adjustment and low retrieval efficiency when the number of objects is large. The specific solution is as follows:

[0007] A Ceph-based distributed object storage retrieval method is applied to a Ceph system, wherein the Ceph system includes: ceph-radosgw, Librados, and rados. The method includes:

[0008] receiving a request for retrieving an object initiated by a client, and determining s_max based on the request for retrieving the object;

[0009] Determine whether the last searched object adjacent to the current searched object is a file or a directory;

[0010] If the last search object is a directory, determine max based on s_max, read the current search object a preset number of times based on max, adjust max to a preset number, and continue reading based on the preset number until all directories in the current layer are read, wherein the current directory is the current directory;

[0011] When the last search object is a file, max is determined based on s_max, max is adjusted based on the number of files of the current search object, and the current search object is read based on max until the current directory is finished, wherein the number of files is positively correlated with max.

[0012] Optionally, the above method includes receiving a request for retrieving an object initiated by a client, and determining s_max based on the request for retrieving an object, including:

[0013] Determining whether the retrieval object request is executed for the first time;

[0014] If so, set the s_max to 1;

[0015] If not, search the global container for the object name in the object search request. If found, use the historical s_max corresponding to the object name as the s_max. If not found, set the s_max to 1.

[0016] Optionally, the above method reads the current search object based on the max value until the directory at this level is completely read, including:

[0017] When the current search object is read, the current search result is stored in the global container, and a preset read identifier is fed back to the client;

[0018] The preset reading identifier is read, and when the preset reading identifier satisfies a preset value, it is determined that the reading of the directory at the current layer is completed.

[0019] The above method, optionally, further comprises mounting the Ceph system through nfs-ganesha, specifically as follows:

[0020] When a client mounted via nfs-ganesha is detected to be performing a 1s operation on a storage directory, the maker value of the retrieval directory is obtained, and s_maxs is determined based on the maker value;

[0021] Call the OSD read object in the rados, and store the obtained search results in the global container;

[0022] After calling back the nfs-ganesha, a lookup request is called to query the metadata information corresponding to the current search object;

[0023] A handle is created based on the metadata information, and the handle is fed back to the client.

[0024] The above method optionally calls a lookup request to query metadata information corresponding to the current search object, including:

[0025] If the current search object is a directory, read the corresponding metadata information in the global container;

[0026] If the current search object is a file, a Get request is called to download metadata information corresponding to the current search object.

[0027] A Ceph-based distributed object storage and retrieval device is applied to a Ceph system, wherein the Ceph system includes: ceph-radosgw, Librados, and rados. The device includes:

[0028] a determination module, configured to receive a retrieval object request initiated by a client, and determine s_max based on the retrieval object request;

[0029] A judgment module, used to judge whether the last search object adjacent to the current search object is a file or a directory;

[0030] a first determination module configured to, when the last search object was a directory, determine max based on s_max, read the current search object a preset number of times based on max, adjust max to a preset number, and continue reading based on the preset number until all directories in the current layer are read, wherein the current directory is the current directory;

[0031] The second determination module is used to determine max based on s_max when the last search object is a file, adjust max based on the number of files of the current search object, and read the current search object based on max until the current layer directory is finished, wherein the number of files is positively correlated with max.

[0032] In the above device, optionally, the determining module includes:

[0033] a judging unit, configured to judge whether the object retrieval request is executed for the first time;

[0034] A first setting unit, configured to set the s_max to 1 if yes;

[0035] The second setting unit is used to search for the object name in the retrieval object request in the global container if not, and if found, use the historical s_max corresponding to the object name as the s_max; if not found, set the s_max to 1.

[0036] In the above device, optionally, the first determination module includes:

[0037] A storage and feedback unit, configured to store the current search result in a global container and feed back a preset reading identifier to the client when the reading of the current search object is completed;

[0038] The reading and determining unit is configured to read the preset reading identifier and determine that the directory at the current layer has been read when the preset reading identifier satisfies a preset value.

[0039] The above device, optionally, further comprises:

[0040] An acquisition and determination module, configured to acquire the maker value of the retrieval directory upon detecting that a client mounted via nfs-ganesha performs a 1s operation in the storage directory, and determine s_maxs based on the maker value;

[0041] A calling and storing module, configured to call the OSD read object in the rados and store the obtained search results in the global container;

[0042] A calling and querying module, configured to call back the nfs-ganesha and then call a lookup request to query metadata information corresponding to the current search object;

[0043] The creation and feedback module is used to create a handle based on the metadata information and feed the handle back to the client.

[0044] In the above device, optionally, the calling and querying module includes:

[0045] a reading unit, configured to read corresponding metadata information in the global container if the current search object is a directory;

[0046] The downloading unit is configured to, if the current search object is a file, call a Get request to download metadata information corresponding to the current search object.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] The present invention discloses a Ceph-based distributed object storage retrieval method and device, which is applied to a Ceph system. The method receives a retrieval object request initiated by a client, determines s_max based on the retrieval object request, determines whether the last retrieval object adjacent to the current retrieval object is a file or a directory, determines max based on s_max if the object is a directory, reads the current retrieval object a preset number of times based on max, adjusts max to a preset number, and continues reading based on the preset number until all directories on the current layer are read; and adjusts max based on the number of files in the current retrieval object, reads the current retrieval object based on max until all directories on the current layer are read, and the number of files is positively correlated with max. The above process distinguishes the type of the current retrieval object, and adopts different max values ​​for flexible adjustment when the object is a file or a directory. This can improve retrieval efficiency when there are many retrieval objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flowchart of a Ceph-based distributed object storage and retrieval method disclosed in an embodiment of the present invention;

[0051] Figure 2 This is a structural block diagram of a Ceph-based distributed object storage and retrieval device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] The present invention discloses a Ceph-based distributed object storage retrieval method and device, which is applied to the retrieval process of objects in the Ceph system. Ceph is a distributed storage system with high performance, high availability, high scalability and rich features, and supports file storage, object storage and block storage. Among them, in object storage, the RGW component (radosgateway) provided by Ceph can support docking with the S3 protocol and the Switch protocol, and the client can perform functions such as uploading / downloading / deleting / retrieval of objects (files) through the protocol. At the same time, Ceph object storage supports the use of nfs-ganesha to mount buckets and access the content of object storage in a manner similar to file storage, which greatly improves the convenience of accessing object storage.

[0054] The basic process of Ceph object storage is as follows:

[0055] There are many types of clients. The most commonly used client for connecting to the S3 protocol is s3cmd. When the client initiates any request, it first reaches the RadosGW module on the Ceph side, that is, the object storage gateway module.

[0056] The RadosGW module will perform request type judgment, various parameter initialization and complex processing, and then call the librados module.

[0057] The librados module is mainly an encapsulation of the underlying Rados object storage system, providing a unified interface to RGW or other modules, and calling into Rados through librados.

[0058] The rados object storage system includes the mon and osd modules, where osd is a daemon process responsible for storing object data / metadata. After the request is obtained from osd, the result is returned to the top-level client through the original path.

[0059] Currently, when searching for subdirectories within a bucket or directory in Ceph object storage using a client such as s3cmd (for example, executing the s3cmd ls command), the search time is normal for a small or moderate number of directories. However, when the number of directories is large, the search takes a very long time, with approximately 1,000 directories requiring around 30 seconds to display results. When mounted on a client using nfs-ganesha and directory searches performed as file storage (for example, executing the ls command), the search time is even longer, and with 1,000 directories, the search becomes almost unusable.

[0060] Based on the above problems, the present invention provides a Ceph distributed object storage retrieval method, which is applied to the Ceph system. The Ceph system includes: ceph-radosgw, Librados and rados. The execution process of the retrieval method is as follows: Figure 1 As shown, the steps include:

[0061] S101, receiving a search object request initiated by a client, and determining s_max based on the search object request;

[0062] In this embodiment of the present invention, a client initiates an s3cmd ls request to retrieve an object, which is a readdir request on the Ceph side. The Ceph client determines whether the request is a first-time execution. The specific determination process is as follows: The first-time execution is determined by checking that the cur_marker field in the retrieval request URL passed in by the client is empty. If this is not the first time execution, cur_marker will be the last result of the previous retrieval.

[0063] If it is the first execution, the readdir request reaches ceph-radosgw, and RGW will create a corresponding Handler based on the client type. When creating it, it passes s_max as 1, creates a List_Bucket object (retrieval object class), and records the s_max of the last object retrieved in this time in the global container. After the List_Bucket object is created, it enters the list_object process.

[0064] If it is not the first execution, it is determined whether the retrieval is complete based on truncate. If the retrieval is complete, it continues the last execution and first searches the global container to see if the object at the end of the last retrieval exists. If found, the readdir request reaches ceph-radosgw, and a Handler is created according to the client type to obtain the s_max value of the last execution and pass it into the Handler. A List_Bucket object (retrieval object class) is created and the s_max of the last object retrieved is recorded in the global container. If the length of the global container exceeds 10, the earliest record is deleted. Furthermore, after the List_Bucket object is created, it enters the list_object process. If not found, it is processed as a new request with s_max set to 1. If the retrieval is not complete, it is processed according to the first execution.

[0065] After the Handler is created, it calls the ListBucket layer to perform specific transactions for object retrieval.

[0066] S102: Determine whether the previous search object adjacent to the current search object is a file or a directory;

[0067] In the embodiment of the present invention, during the search process, a maximum number of reads is set, wherein the maximum number of reads can be set based on experience or specific circumstances. In the embodiment of the present invention, no specific limitation is imposed. Preferably, the embodiment of the present invention takes the maximum number of reads as 51 as an example for explanation. After setting the maximum number of reads, it is determined whether attempt is 1. If so, an object read operation is performed with max=1, save_max=max, attempt++, and the operation is performed in the Rados layer and finally in the OSD layer. If not, it is determined whether the current search object is a file or a directory. The specific determination process is as follows:

[0068] The directory situation when reading can be determined by whether the read result ends with " / ". If it ends with " / ", it means it is a directory. If it does not end with " / ", it means it is not a directory, that is, a file.

[0069] S103, if the last search object was a directory, determining max based on s_max, reading the current search object a preset number of times based on max, adjusting max to a preset number, and continuing to read based on the preset number until all directories in the current layer are read, wherein the current directory is the current directory;

[0070] In the embodiment of the present invention, if the last search object is a directory, max is always 1. If attempt is less than 31, the object is read with max = 1, save_max = max, max_items = 1000, attempt++, and then the object is read from the Rados layer and finally read from the OSD. After attempt > 31 (which can be modified by the daemon), max is changed to 300000, save_max = max, max_items = 1000, attempt++, and finally read from the Rados layer and finally read from the OSD.

[0071] After the search is complete, the results are recorded in the result and the last result is recorded in the global container. If the reading is not complete, the truncate value is set to 1 and the directory is returned to the upper layer until it reaches the client. The client will send a new readdir request and continue traversing the previous result. This process repeats until the directory at this layer is completely read and the truncate value is set to 0, indicating the end of the search.

[0072] S104. When the last search object is a file, determine max based on s_max, adjust max based on the number of files of the current search object, and read the current search object based on max until the current directory is finished, wherein the number of files is positively correlated with max.

[0073] In the embodiment of the present invention, when the last search object is a file, max will increase as the number of files increases, from 1->100->1000->10000. When s_max=1, the object is read, max=100, save_max=max, max_items=1000, attempt++, and then it is read in the Rados layer and finally in the OSD for reading. When s_max=100, the object is read, max=1000, save_max=max, max_items=1000, attempt++, and then it is read in the Rados layer and finally in the OSD for reading. When s_max=1000, the object is read, max=10000, save_max=max, max_items=1000, attempt++, and then it is read in the Rados layer and finally in the OSD for reading.

[0074] After the search is complete, the results are recorded in the result and the last result is recorded in the global container. If the reading is not complete, the truncate value is set to 1 and the directory is returned to the upper layer until it reaches the client. The client will send a new readdir request and continue traversing the previous result. This process repeats until the directory at this layer is completely read and the truncate value is set to 0, indicating the end of the search.

[0075] The present invention discloses a Ceph-based distributed object storage retrieval method, which is applied to a Ceph system and receives a retrieval object request initiated by a client. Based on the retrieval object request, s_max is determined; the last retrieval object adjacent to the current retrieval object is determined to be a file or a directory; if the object is a directory, max is determined based on s_max, the current retrieval object is read a preset number of times based on max, max is adjusted to a preset number, and reading continues based on the preset number until all directories on the current layer are read; if the object is a file, max is adjusted based on the number of files in the current retrieval object, and the current retrieval object is read based on max until all directories on the current layer are read, wherein the number of files is positively correlated with max. The above process distinguishes the type of the current retrieval object and flexibly adjusts the max value when the object is a file or a directory. This can improve retrieval efficiency when there are many retrieval objects.

[0076] In the embodiment of the present invention, in the prior art, the specific processing process of nfs-ganesha docking with Ceph object storage transformation is as follows:

[0077] a1. When a Linux client mounted using nfs-ganesha performs an ls operation on an object storage directory, nfs-ganesha receives the request and, through thread pool scheduling, first enters the nfs4_Compound process and then proceeds to fsal_readdir to perform a series of initialization and accessibility checks. This initialization process includes verifying various NFS parameters and extracting and assigning values ​​to various variables. Accessibility checks typically involve checking access rights on the server being accessed, such as read / write permissions and blacklisting. Only requests with appropriate permissions are processed and sent to the server. After completion, in the initial search, a call is ultimately made to ceph-radosgw.

[0078] b1. Follow the rgw_readdir2 process and pass in the callback function to query and create a handle for the final result.

[0079] c1. In the rgw_readdir2 process, the first step is to determine whether the current read object is in the bucket directory (i.e., the root directory). The specific determination process is as follows: Determine whether it is the bucket directory, that is, whether the current directory is a mounted directory. For example, if the mounted directory is / Bucket1, then if readdir retrieves s3: / / Bucket1, then it is querying the bucket directory. If so, the bucket query is performed. If not, a new RGWReaddirRequest object is created, which is a subclass of the ceph-radosgw search process described above. It enters the execute_reg function to perform a series of initialization and verification operations.

[0080] d1. Enter the list_object process, where the list_object process is the process in the Ceph-RadosGW end transformation. That is to say, the transformation of nfs-ganesha to Ceph object storage is based on the transformation of the first part. Call the ceph-radosgw object retrieval process described above to read the object from the OSD. After the retrieval is completed, the result is in the result. All the retrieval results will be called back one by one to determine whether result.count is greater than 0. If not, return the handle to nfs-ganesha. Ganesha determines whether the retrieval is completed. If so, enter nfs-ganesha. After receiving the request, it will enter the nfs4_Compound process first through thread pool scheduling. If not, call the ceph-radosgw end

[0081] e1. If result.count is greater than 0, a response is sent to nfs-ganesha. After the callback is returned to nfs-ganesha, ganesha calls the lookup request to query the metadata information of the search object. The lookup request returns to the ceph-radosgw end again. If it is a directory, it calls list_object again to search the OSD again, and then extracts the detailed metadata information and returns it to nfs-ganesha. If it is a file, it calls RGWGet request to download the object and returns the downloaded object metadata information to nfs-ganesha.

[0082] f1.ganesha creates a handle based on these metadata. At the same time, ganesha will determine whether the traversal is complete. If not, it will send a Readdir request to ceph-radosgw again and continue the search based on the last result. If the traversal is complete, the handle will be returned to the Linux client, and the client will finally display it to the user in the form of a file system.

[0083] In the embodiment of the present invention, the process of initiating a search request when the nfs-ganesha is connected to the ceph object storage is modified, and the lookup process in the above processing is improved. The improved solution is as follows:

[0084] When a Linux client mounted using nfs-ganesha performs an ls operation on an object storage directory, nfs-ganesha receives the request and, through a thread pool, first enters the nfs4_Compound process and then proceeds to fsal_readdir to perform a series of initialization and accessibility checks. This initialization process includes verifying various NFS parameters and extracting and assigning values ​​to various variables. Accessibility checks typically involve checking access rights on the server being accessed, such as read / write permissions and blacklisting. Only requests with appropriate permissions proceed to the server for processing. After completion, in the initial search, a call is ultimately made to ceph-radosgw.

[0085] b2. Follow the rgw_readdir2 process and pass in the callback function to query and create a handle for the final result.

[0086] c1. In the rgw_readdir2 process, the read object is first determined to be in the bucket directory (i.e., the root directory). The specific determination process is as follows: Determine whether it is the bucket's search directory, that is, whether the current search directory is a mounted directory. For example, if the mounted directory is / Bucket1, then if readdir retrieves s3: / / Bucket1, it is the bucket's search directory. If so, the bucket query is performed. Otherwise, the maker function determines whether this is the first search. If maker is not 0, it is not the first search, meaning it is a subsequent search. The global container is first searched for the object name from the previous search. If it is found, it is determined to be a directory or file. For directories, s_max remains 1; for files, s_max is the value of the previous s_max. If maker is 0, it is determined to be the first search, and s_max is set to 1. A new RGWReaddirRequest object is created, which is a subclass of the ceph-radosgw search process described above. The execute function executes excute_reg to perform a series of initialization and verification operations.

[0087] d2. Enters the list_object process and calls the same ceph-radosgw object search process described above to read objects from the OSD. However, the search logic is modified to the previously described search logic. After the search is complete, all search results are processed one by one. After the search, the results are stored in the global list, and the last result is stored in the global container.

[0088] After calling back to nfs-ganesha, ganesha calls a lookup request to query the metadata information of the search object. If the search result is a directory, ganesha does not search the OSD again. Instead, it directly retrieves the result recorded in the global list, extracts the metadata locally, and returns the handle to nfs-ganesha, saving a lot of time for further search. The file is unchanged because the file lookup is actually a Get request. After the previous object search operation, the object has been cached and does not need to be read from the OSD, so the file does not need to be modified.

[0089] f2.ganesha creates a handle based on these metadata. At the same time, ganesha will use truncate to determine whether the traversal is complete. If the traversal is not complete, it will send a Readdir request to ceph-radosgw again and continue the search based on the last result. If the traversal is complete, the handle will be returned to the Linux client, and the client will finally display it to the user in the form of a file system.

[0090] In the embodiment of the present invention, with respect to the above-mentioned processing, during the modification of the Ceph-RadosGW side, the original search process does not distinguish between files and directories, and the number of objects read from the OSD is fixed, resulting in an inability to achieve flexible adjustment. Therefore, when the number of objects is large, the client experience is poor. In the modified search process, when the number of directories is small or moderate, the reading of redundant objects can be reduced to reduce the time consumed for a single read; when the number of directories is large, the number of interactions with the OSD and the client can be reduced to reduce the additional processing and network communication time caused by excessive interactions; when the number of files is large, the number of files read at one time can be increased to reduce the interaction between the rgw, the OSD, and the client.

[0091] In the nfs-ganesha integration with Ceph object storage, the core search process was replaced with the modified one, significantly improving object retrieval speed. Furthermore, handle creation was performed by eliminating the need for repeated searches and instead using the results of the first search, significantly improving processing efficiency.

[0092] Based on the above-mentioned Ceph-based distributed object storage retrieval method, an embodiment of the present invention further provides a Ceph-based distributed object storage retrieval device, which is applied to a Ceph system. The Ceph system includes: ceph-radosgw, Librados and rados. The structural block diagram of the device is as follows: Figure 2 As shown, including:

[0093] Determination module 201 , judgment module 202 , first judgment module 203 and second judgment module 204 .

[0094] in,

[0095] The determining module 201 is configured to receive a search object request initiated by a client and determine s_max based on the search object request;

[0096] The judgment module 202 is used to judge whether the last search object adjacent to the current search object is a file or a directory;

[0097] The first determination module 203 is configured to, when the last search object was a directory, determine max based on s_max, read the current search object a preset number of times based on max, adjust max to a preset number, and continue reading based on the preset number until all directories in the current layer are read, wherein the current directory is the current directory;

[0098] The second determination module 204 is used to determine max based on s_max when the last search object is a file, adjust max based on the number of files in the current search object, and read the current search object based on max until the current directory is finished, wherein the number of files is positively correlated with max.

[0099] The present invention discloses a Ceph-based distributed object storage retrieval method and device, which is applied to a Ceph system. The method receives a retrieval object request initiated by a client, determines s_max based on the retrieval object request, determines whether the last retrieval object adjacent to the current retrieval object is a file or a directory, determines max based on s_max if the object is a directory, reads the current retrieval object a preset number of times based on max, adjusts max to a preset number, and continues reading based on the preset number until all directories on the current layer are read; and adjusts max based on the number of files in the current retrieval object, reads the current retrieval object based on max until all directories on the current layer are read, and the number of files is positively correlated with max. The above process distinguishes the type of the current retrieval object, and adopts different max values ​​for flexible adjustment when the object is a file or a directory. This can improve retrieval efficiency when there are many retrieval objects.

[0100] In the embodiment of the present invention, the determining module 201 includes:

[0101] A judging unit 205 , a first setting unit 206 and a second setting unit 207 .

[0102] in,

[0103] The judging unit 205 is configured to judge whether the object retrieval request is executed for the first time;

[0104] The first setting unit 206 is configured to set the s_max to 1 if yes;

[0105] The second setting unit 207 is configured to search for the object name in the object search request in a global container if not found, and use the historical s_max corresponding to the object name as the s_max if found; otherwise, set the s_max to 1.

[0106] In the embodiment of the present invention, the first determination module 203 includes:

[0107] Retrieval and feedback unit 208 and reading and decision unit 209.

[0108] in,

[0109] The storage and feedback unit 208 is used to store the current search result in the global container when the current search object is read, and to feed back a preset reading identifier to the client;

[0110] The reading and determining unit 209 is configured to read the preset reading identifier and determine that the reading of the directory at the current layer is complete when the preset reading identifier satisfies a preset value.

[0111] In an embodiment of the present invention, the device further includes:

[0112] Acquisition and determination module 210, calling and storage module 211, calling and query module 212 and creation and feedback module 212.

[0113] in,

[0114] The acquisition and determination module 210 is configured to, upon detecting that a client mounted via nfs-ganesha performs a 1s operation in a storage directory, acquire a maker value of the retrieval directory and determine s_maxs based on the maker value;

[0115] The calling and storing module 211 is used to call the OSD read object in the rados and store the obtained search results in the global container;

[0116] The calling and querying module 212 is used to call back the nfs-ganesha and then call a lookup request to query the metadata information corresponding to the current search object;

[0117] The creation and feedback module 213 is configured to create a handle based on the metadata information and feed the handle back to the client.

[0118] In the embodiment of the present invention, the calling and querying module 212 includes:

[0119] Reading unit 214 and downloading unit 215 .

[0120] in,

[0121] The reading unit 214 is configured to read corresponding metadata information in the global container if the current search object is a directory;

[0122] The download unit 216 is configured to, if the current search object is a file, invoke a Get request to download metadata information corresponding to the current search object. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referenced for reference. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple. For relevant details, refer to the description of the method embodiments.

[0123] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0124] The above is a detailed introduction to a Ceph-based distributed object storage and retrieval method and device provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A Ceph-based distributed object storage retrieval method, characterized in that: Applied to a Ceph system, the Ceph system includes: ceph-radosgw, Librados, and rados, the method includes: receiving a request for retrieving an object initiated by a client, and determining s_max based on the request for retrieving the object; Determine whether the last searched object adjacent to the current searched object is a file or a directory; If the last search object is a directory, determine max based on s_max, read the current search object a preset number of times based on max, adjust max to a preset number, and continue reading based on the preset number until all directories in the current layer are read, wherein the current directory is the current directory; In the case where the last search object is a file, max is determined based on s_max, max is adjusted based on the number of files in the current search object, and the current search object is read based on max until all directories in the current layer are read, wherein the number of files is positively correlated with max; The step of receiving a request for retrieving an object initiated by a client and determining s_max based on the request for retrieving the object includes: Determining whether the retrieval object request is executed for the first time; If so, set the s_max to 1; If not, search the global container for the object name in the object search request. If found, use the historical s_max corresponding to the object name as the s_max. If not found, set the s_max to 1.

2. The method according to claim 1, characterized in that Reading the current search object based on the max until the directory at this level is completely read includes: When the current search object is read, the current search result is stored in the global container, and a preset read identifier is fed back to the client; The preset reading identifier is read, and when the preset reading identifier satisfies a preset value, it is determined that the reading of the directory at the current layer is completed.

3. The method according to claim 1, characterized in that The method further includes mounting the Ceph system through nfs-ganesha, specifically as follows: When a client mounted via nfs-ganesha is detected to be performing a 1s operation on a storage directory, the maker value of the retrieval directory is obtained, and s_maxs is determined based on the maker value; Call the OSD read object in the rados, and store the obtained search results in the global container; After calling back the nfs-ganesha, a lookup request is called to query the metadata information corresponding to the current search object; A handle is created based on the metadata information, and the handle is fed back to the client.

4. The method according to claim 3, characterized in that Call a lookup request to query the metadata information corresponding to the current search object, including: If the current search object is a directory, read the corresponding metadata information in the global container; If the current search object is a file, a Get request is called to download metadata information corresponding to the current search object.

5. A Ceph-based distributed object storage and retrieval device, characterized in that: Applied to a Ceph system, the Ceph system includes: ceph-radosgw, Librados and rados, and the device includes: a determination module, configured to receive a retrieval object request initiated by a client, and determine s_max based on the retrieval object request; A judgment module, used to judge whether the last search object adjacent to the current search object is a file or a directory; a first determination module configured to, when the last search object was a directory, determine max based on s_max, read the current search object a preset number of times based on max, adjust max to a preset number, and continue reading based on the preset number until all directories in the current layer are read, wherein the current directory is the current directory; a second determination module configured to, when the last search object was a file, determine max based on s_max, adjust max based on the number of files in the current search object, and read the current search object based on max until all directories in the current layer are read, wherein the number of files is positively correlated with max; Wherein, the determination module includes: a judging unit, configured to judge whether the object retrieval request is executed for the first time; A first setting unit, configured to set the s_max to 1 if yes; The second setting unit is used to search for the object name in the retrieval object request in the global container if not, and if found, use the historical s_max corresponding to the object name as the s_max; if not found, set the s_max to 1.

6. The device according to claim 5, characterized in that The first determination module includes: A storage and feedback unit, configured to store the current search result in a global container and feed back a preset reading identifier to the client when the reading of the current search object is completed; The reading and determining unit is configured to read the preset reading identifier and determine that the directory at the current layer has been read when the preset reading identifier satisfies a preset value.

7. The device according to claim 5, characterized in that The apparatus further comprises: an acquisition and determination module for acquiring a maker value of a retrieval directory upon detecting that a client mounted via nfs-ganesha performs a 1s operation in a storage directory, and determining s_maxs based on the maker value; A calling and storing module, configured to call the OSD read object in the rados and store the obtained search results in the global container; A calling and querying module, configured to call back the nfs-ganesha and then call a lookup request to query metadata information corresponding to the current search object; The creation and feedback module is used to create a handle based on the metadata information and feed the handle back to the client.

8. The device according to claim 7, characterized in that The calling and querying module includes: a reading unit, configured to read corresponding metadata information in the global container if the current search object is a directory; The downloading unit is configured to, if the current search object is a file, call a Get request to download metadata information corresponding to the current search object.

Images