Distributed object storage method, device, system and metadata server

Abstract

The present application relates to a distributed object storage method, device, system and metadata server, the method comprising: when receiving an object application request initiated by a client, obtaining a writable area of ​​an SMR disk in a distributed storage system; wherein the object application request includes the size of a file to be written, and the SMR disk is divided into a plurality of writable areas; determining write information according to the object application request and the remaining storage space of the writable area; the write information includes the length of each object obtained by dividing the file to be written; returning the write information to the client, so that the client writes the file to be written to a corresponding storage node according to the write information. Through the present application, the aggregate storage management of the Zone space is realized in the case of variable object length, which can maximize the use of the Zone space and reduce the waste of storage space.

Description

Technical Field

[0001] The present application relates to the technical field of distributed object storage, and in particular to a distributed object storage method, device, system and metadata server. Background Art

[0002] With the continuous development of storage technology, object-based storage (OBS) has emerged to meet the ever-changing storage needs of network cluster systems. Object-based storage is a new type of storage structure, based on which a distributed object storage file system can be established, with cross-platform data sharing, fine-grained application data management, high performance and good scalability.

[0003] In a distributed file system, when a user writes a file, the file is divided into objects of fixed size and a fixed space size is allocated. Generally, the object length is defined to be relatively large, such as Figure 1 As shown in the figure, writing a 400MB file requires 4 objects for storage, and the objects are often not fully written. At this time, if the object is stored on a traditional CMR disk, there will not be much problem. However, when storing based on an SMR disk (Shingled Magnetic Recording), since the minimum granularity of SMR disk management is the storage area Zone, the fixed size of the Zone is 256MB, and only appending is allowed within the Zone. To delete data, the entire Zone must be erased. Erasing partial data and random writing are not allowed. If writing is performed based on the existing method, each object is written to a Zone, and the failure of the Zone to be fully written will result in a very large waste of space.

[0004] To address the serious space waste problem, in the prior art, when writing files, zones are reused, unfilled zones are filled, and then the next zone is switched. The file storage diagram at this time is as follows: Figure 2 As shown in the figure, the space waste for writing a 400MB file has been greatly improved, but there is still 112MB of space wasted. Furthermore, if the remaining unused 112MB of space in the zone also needs to be reused, in the current distributed storage system, the object size is fixed, and there is only 112MB of space at this time, which cannot accommodate a complete object. If an object is stored in two different zones, it will cause great trouble for subsequent life cycle management, data recovery and metadata management. Summary of the invention

[0005] The embodiments of the present application provide a distributed object storage method, device, system and metadata server to at least solve the problem of space waste during object storage in the related art.

[0006] In a first aspect, an embodiment of the present application provides a distributed object storage method, including:

[0007] When receiving an object application request initiated by a client, obtaining a writable area of ​​an SMR disk in a distributed storage system; wherein the object application request includes the size of a file to be written, and the SMR disk is divided into a plurality of writable areas;

[0008] Determine writing information according to the object application request and the remaining storage space of the writable area; the writing information includes the length of each object obtained by segmenting the file to be written;

[0009] The write information is returned to the client, so that the client writes the file to be written into the corresponding storage node according to the write information.

[0010] In some embodiments, determining the write information according to the object application request and the remaining storage space of the writable area includes:

[0011] Determining a plurality of first objects according to the object application request and a preset object length;

[0012] Acquire the remaining storage space of the writable area, and determine the adjustment information of the object length according to the remaining storage space;

[0013] Adjust the size and length of the first object according to the adjustment information to obtain a second object;

[0014] Capacity space is pre-allocated according to the length of the second object, a writable area is allocated according to the pre-allocated capacity space, and writing information corresponding to the file to be written is obtained.

[0015] In some embodiments, obtaining the remaining storage space of the writable area and determining the adjustment information of the object length according to the remaining storage space includes:

[0016] Acquire the remaining storage space of the writable area;

[0017] When the remaining storage space of the writable area is smaller than the object length, the object length adjustment information is determined to be a reduction in the object length; wherein the reduced object length is not smaller than the data size in the object.

[0018] In some embodiments, when the remaining storage space of the writable area is less than the object length, the adjustment information for determining the object length is:

[0019] The object length is reduced to the size of the data within the object.

[0020] In some embodiments, pre-allocating capacity space according to the length of the second object, allocating a writable area according to the pre-allocated capacity space, and obtaining writing information corresponding to the file to be written includes:

[0021] Get at least one writable area;

[0022] Acquire the corresponding pre-allocated capacity space according to the length of each of the second objects;

[0023] Writing the corresponding second objects into the remaining storage space of the writable area in sequence according to the pre-allocated capacity space;

[0024] According to the writable area to be written, writing information corresponding to the file to be written is determined; index information of the writable area and the length of each second object in the writable area.

[0025] In some embodiments, after returning the write information to the client so that the client writes the to-be-written file into the corresponding storage node according to the write information, the process further includes:

[0026] The remaining storage space of the writable area uploaded by the storage node is obtained; the remaining storage space is calculated by the storage node based on the pre-allocated capacity space of at least one second object in the writable area after the file to be written is written into the storage node.

[0027] In some embodiments, after returning the write information to the client so that the client writes the to-be-written file into the corresponding storage node according to the write information, the process further includes:

[0028] Get the length of each second object actually written into the writable area;

[0029] When the length of the second object actually written into the writable area is smaller than the corresponding pre-allocated capacity space, the pre-allocated capacity space is corrected to be the length of the second object actually written into the writable area.

[0030] In some embodiments, the written information further includes the number of objects in the writable area, and the method further includes:

[0031] Persistently storing the written information;

[0032] When the number of objects in the writable area is zero, the corresponding writable area is deleted.

[0033] In a second aspect, an embodiment of the present application provides a distributed object storage device, including:

[0034] A writable area acquisition unit, configured to acquire a writable area of ​​an SMR disk in a distributed storage system when receiving an object application request initiated by a client; wherein the object application request includes the size of a file to be written, and the SMR disk is divided into a plurality of writable areas;

[0035] A writing information determining unit, configured to determine writing information according to the object application request and the remaining storage space of the writable area; the writing information includes the length of each object obtained by segmenting the file to be written;

[0036] The storage unit is used to return the write information to the client, so that the client writes the file to be written into the corresponding storage node according to the write information.

[0037] In a third aspect, an embodiment of the present application provides a metadata server, comprising a memory and a processor, characterized in that a computer program is stored in the memory, and when the processor executes the computer program, the distributed object storage method described in the first aspect above is implemented.

[0038] In a fourth aspect, an embodiment of the present application provides a distributed object storage system, including: a client, at least one storage node, and a metadata server; wherein the metadata server is in communication connection with each storage node and the client;

[0039] The client is used to initiate an object application request to the metadata server, and write the file to be written into the corresponding storage node according to the writing information returned by the metadata server;

[0040] The storage node is used to store the files to be written by the client;

[0041] The metadata server is used to implement the distributed object storage method as described in the first aspect above.

[0042] In some embodiments, the storage node is further configured to:

[0043] The remaining storage space of the corresponding writable area is calculated based on the pre-allocated capacity space of the second object in each writable area, and the remaining storage space of the writable area is uploaded to the metadata server.

[0044] Compared with the related art, the distributed object storage method provided in the embodiment of the present application determines the writing information according to the object application request and the remaining storage space of the writable area. The writing information includes the length of each object obtained by dividing the file to be written, thereby determining the object length according to the writing request and the remaining storage space of the writable area, making the object length variable and facilitating the space management of the object. By returning the writing information to the client, the client writes the file to be written to the corresponding storage node according to the writing information. In the case where the object length is variable, the aggregate storage management of the Zone space is realized, which can maximize the use of the Zone space and reduce the waste of storage space.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 It is a file storage schematic diagram in the prior art;

[0048] Figure 2 It is another file storage schematic diagram in the prior art;

[0049] Figure 3 This is a schematic diagram of an application scenario of a distributed object storage method in one of the embodiments of the present application;

[0050] Figure 4 It is a flowchart of a distributed object storage method in one of the embodiments of the present application;

[0051] Figure 5 This is a schematic diagram of the storage of file 1 and file 2 in one embodiment of the present application;

[0052] Figure 6 This is a schematic diagram of a process for determining write information in one embodiment of the present application;

[0053] Figure 7 This is a structural block diagram of a distributed object storage device in one of the embodiments of the present application;

[0054] Figure 8 This is a schematic diagram of the structure of a metadata server in one of the embodiments of the present application;

[0055] Fig. 9It is a structural block diagram of a distributed object storage system in one of the embodiments of the present application.

[0056] Description of the drawings: 101, metadata server; 102, storage node; 103, client; 701, writable area acquisition unit; 702, write information determination unit; 703, storage unit; 80, bus; 81, processor; 82, memory; 83, communication interface. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0058] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0059] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0060] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0061] Distributed File System (DFS) refers to a file system that manages physical storage resources that are not necessarily directly connected to local nodes, but are connected to nodes through a computer network; or a complete hierarchical file system formed by combining several different logical disk partitions or volumes. Distributed file systems can effectively solve the problem of data storage and management, and expand a file system fixed to a certain location to any number of locations / multiple file systems. Numerous nodes form a file system network. Each node can be distributed in different locations, and communication and data transmission between nodes can be carried out through the network.

[0062] SMR (Shingled Magnetic Recording) disk is a high-capacity disk that uses a new type of magnetic storage technology. SMR disks partially overlap the data tracks on the disk, just like tiles on a roof. This technology is called shingled magnetic recording technology. The technology has very small changes in the manufacturing process, but it can greatly increase the storage density of the disk and effectively reduce the storage cost per unit capacity of the disk.

[0063] The distributed object storage method provided in this application can be applied to a distributed file system based on SMR disks, such as Figure 3The following is a schematic diagram of an application scenario of the distributed object storage method provided in the embodiment of the present application. Figure 3 As shown, the metadata server is in communication connection with each storage node and the client. The client sends an object application request to the metadata server. The metadata server allocates storage information based on the object application request and the writable area of ​​each communication node and returns it to the client. The client requests and completes the write operation from the corresponding storage node based on the storage information.

[0064] This embodiment provides a distributed object storage method, which can be executed by a metadata server in a distributed object storage system. Figure 4 is a flow chart of a distributed object storage method according to an embodiment of the present application. Figure 4 As shown, the process includes the following steps:

[0065] Step S101, when an object application request initiated by a client is received, a writable area of ​​an SMR disk in a distributed storage system is obtained; wherein the object application request includes the size of a file to be written, and the SMR disk is divided into multiple writable areas.

[0066] An object is the basic unit of data storage, consisting of metadata, user data, and file names. In a storage node of a distributed storage system, an SMR disk can be divided into multiple writable zones. The space management of an SMR disk is based on the size of a zone (e.g., 256MB), and a zone is used to store objects.

[0067] In this embodiment, when a client requests to store a file, it is necessary to divide the file to be written into multiple objects, and then store the objects in the writable area Zone of the SMR disk in the distributed object storage system. Specifically, the client first generates an object application request based on the file to be written, and then initiates the object application request to the metadata server. When the metadata server receives the object application request initiated by the client, it obtains the writable area Zone of the SMR disk in the distributed storage system to perform subsequent storage space allocation based on the writable area Zone. Optionally, the client can be a desktop computer, a mobile phone, a PDA, etc.

[0068] In this embodiment, the object application request at least includes the size of the file to be written. Of course, in other embodiments, the object application request may also include the device identification of the client or other custom information, which is not specifically limited in this application.

[0069] Step S102, determining writing information according to the object application request and the remaining storage space of the writable area; the writing information includes the length of each object obtained by segmenting the file to be written.

[0070] In the related art, the length of each object obtained by dividing the file to be written is fixed, so the object is often not fully written or the writable area Zone is not fully written. In the context of exponential growth of data, there is a huge waste of space in each storage node.

[0071] In this embodiment, in order to improve the space utilization of the writable area Zone, the write information can be determined according to the object application request and the remaining storage space of the writable area. Specifically, the size of the file to be written can be determined according to the object application request, and the length of each object obtained by dividing the file to be written can be adaptively configured according to the size of the file to be written and the remaining storage space of the writable area, and space management is performed on objects of variable length to maximize the use of Zone space. It can be understood that the write information includes the length of each object obtained by dividing the file to be written, and also includes the index information necessary for writing the file to be written into the storage, including node ID, Zone ID, object ID, etc. This application does not make specific limitations here.

[0072] For example, Figure 5 As shown, when writing a 400MB file 1, the length of each object obtained by adaptively configuring the size of the file to be written and the remaining storage space of the writable area is divided into the file to be written, and 4 objects are required for storage, including object 1 (128MB), object 2 (128MB), object 3 (128MB), and object 4 (16MB); when writing a 368MB file 2, the length of each object obtained by adaptively configuring the size of the file to be written and the remaining storage space of the writable area is divided into the file to be written, and 3 objects are required for storage, including object 1 (112MB), object 2 (128MB), and object 3 (128MB). When writing, file 1 object 1 and file 1 object 2 can be written to the first zone, file 1 object 3, file 1 object 4 and file 2 object 1 can be written to the second zone, and file 2 object 2 and file 2 object 3 can be written to the third zone, thereby realizing the aggregate storage of objects by adaptively configuring the length of each object obtained by adaptively configuring the size of the file to be written and the remaining storage space of the writable area.

[0073] Step S103: Return the write information to the client, so that the client writes the file to be written into the corresponding storage node according to the write information.

[0074] Specifically, in this embodiment, after determining the write information corresponding to the file to be written, the metadata server returns the write information to the client. After receiving the write information, the client sends a write request to the corresponding storage node according to the write information. When the storage node receives the write request, it writes the file to be written into the corresponding writable area.

[0075] In summary, the distributed object storage method provided in the embodiment of the present application determines the write information according to the object application request and the remaining storage space of the writable area, and the write information includes the length of each object obtained by dividing the file to be written, thereby realizing the determination of the object length according to the write request and the remaining storage space of the writable area, making the object length variable and facilitating the space management of the object. By returning the write information to the client, the client writes the file to be written to the corresponding storage node according to the write information. In the case where the object length is variable, the aggregate storage management of the Zone space is realized, which can maximize the use of the Zone space and reduce space waste.

[0076] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0077] like Figure 6 As shown, based on the above embodiments, in some of the embodiments, determining the write information according to the object application request and the remaining storage space of the writable area includes:

[0078] Step S1021: determining a plurality of first objects according to the object application request and a preset object length.

[0079] Specifically, in this embodiment, firstly, the file to be written is obtained according to the object application request, and then the file to be written is divided according to the preset object length to obtain multiple first objects. Among them, the first objects are obtained by dividing according to the preset object length, and the sizes of the first objects are the same. Limited by the memory, CPU, disk and other limitations of the device where the metadata service is located, in order to minimize the amount of metadata and increase the amount of data storage, the preset object length is generally defined to be relatively large, such as 64MB, 128MB, 256MB, etc. The preset object length can be customized, and this application does not make specific restrictions here.

[0080] Step S1022, obtaining the remaining storage space of the writable area, and determining the adjustment information of the object length according to the remaining storage space.

[0081] Step S1023: adjusting the size and length of the first object according to the adjustment information to obtain a second object.

[0082] In this embodiment, the object length adjustment information can be dynamically configured according to the remaining storage space of the writable area, and the size length of the first object can be adjusted according to the adjustment information. Then, the writable area Zone is reused during the writing process, and the second object with adjusted length is aggregated and stored in a writable area Zone, thereby improving the data storage capacity of the writable area.

[0083] Specifically, obtaining the remaining storage space of the writable area, and determining the adjustment information of the object length based on the remaining storage space includes: obtaining the remaining storage space of the writable area; when the remaining storage space of the writable area is less than the object length, determining the adjustment information of the object length is to reduce the object length. In some embodiments, the adjustment information of the object length is that the reduced object length is less than the length of the first object and greater than the data size within the object. In other embodiments, the adjustment information of the object length is: reducing the object length to the data size within the object. The adjustment information of the object length can be customized and is not specifically limited in this application.

[0084] Step S1024: pre-allocate capacity space according to the length of the second object, allocate a writable area according to the pre-allocated capacity space, and obtain write information corresponding to the file to be written.

[0085] In some specific implementations, at least one writable area is first obtained, and the corresponding pre-allocated capacity space is obtained according to the length of each second object, so as to implement pre-allocation of storage space according to the second object after the length adjustment. The pre-allocated capacity space is not less than the length of the corresponding second object. For example, in some embodiments, the size of the pre-allocated capacity space can be consistent with the length of the corresponding second object.

[0086] Then, a writable area is allocated according to the pre-allocated capacity space of each second object, and the corresponding second objects are written into the remaining storage space of the writable area in sequence according to the pre-allocated capacity space; in some embodiments, the writable area Zone corresponding to the second object can be randomly allocated and written sequentially, and when a writable area Zone is full or cannot continue to be written, continue to write to another writable area Zone.

[0087] Finally, the writing information corresponding to the file to be written is determined according to the written writable area. The writing information is the index information of the writable area and the length of each second object in the writable area. The index information can be, for example, the writable area ID and the second object ID in the writable area.

[0088] Based on the above embodiments, in some of the embodiments, after returning the write information to the client so that the client writes the to-be-written file into the corresponding storage node according to the write information, the process further includes:

[0089] The remaining storage space of the writable area uploaded by the storage node is obtained; the remaining storage space is calculated by the storage node based on the pre-allocated capacity space of at least one second object in the writable area after the file to be written is written into the storage node.

[0090] In this embodiment, when the file to be written is written, the remaining storage space of the corresponding writable area is calculated and updated according to the pre-allocated capacity space of the second object. If the remaining storage space after the update is not zero, the writable area is reported to the metadata server for caching. Therefore, when the client initiates an object application request again, the object length and other writing information can be determined according to the writable area.

[0091] Based on the above embodiments, in some of the embodiments, after returning the write information to the client so that the client writes the to-be-written file into the corresponding storage node according to the write information, the process further includes:

[0092] The length of each second object actually written into the writable area is obtained, and when the length of the second object actually written into the writable area is less than the corresponding pre-allocated capacity space, the pre-allocated capacity space is corrected to be the length of the second object actually written into the writable area.

[0093] In this embodiment, there may be a deviation between the pre-allocated capacity space according to the length of the second object and the length of the second object actually written to the writable area, so after the file to be written is written to the corresponding storage node, space management needs to be challenged. Specifically, after the file to be written is written, the length of the second object actually written to the writable area is compared with the corresponding pre-allocated capacity space. When the length of the second object actually written to the writable area is less than the corresponding pre-allocated capacity space, the pre-allocated capacity space needs to be corrected to the length of the second object actually written to the writable area to ensure accurate capacity calculation.

[0094] Based on the above embodiments, in some of the embodiments, the write information also includes the number of objects in the writable area, and the method further includes: persistently storing the write information, and when the number of objects in the writable area is zero, deleting the corresponding writable area.

[0095] In this embodiment, when the writing of the file to be written is completed, the written information is persistently stored to ensure the reliability of the distributed object storage system. In particular, the written information also includes the number of objects in the writable area. When the number of objects in the writable area is zero, the corresponding writable area is deleted. Specifically, when an object is deleted, if an object cannot occupy a zone exclusively, the space cannot be released. It is necessary to wait until all objects are deleted before the space can be released and the corresponding writable area can be deleted.

[0096] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0097] This embodiment also provides a distributed object storage device, which is used to implement the above embodiments and preferred implementations, and will not be repeated for what has been described. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0098] Figure 7 is a structural block diagram of a distributed object storage device according to an embodiment of the present application, such as Figure 7 As shown, the device includes: a writable area acquisition unit 701, a write information determination unit 702 and a storage unit 703.

[0099] The writable area acquisition unit 701 is used to acquire the writable area of ​​the SMR disk in the distributed storage system when receiving an object application request initiated by a client; wherein the object application request includes the size of the file to be written, and the SMR disk is divided into multiple writable areas;

[0100] A writing information determining unit 702 is used to determine writing information according to the object application request and the remaining storage space of the writable area; the writing information includes the length of each object obtained by segmenting the file to be written;

[0101] The storage unit 703 is used to return the write information to the client, so that the client writes the file to be written into the corresponding storage node according to the write information.

[0102] In some embodiments, the write information determination unit 702 includes: a first object determination module, an adjustment information determination module, a second object determination module, and a writable area allocation module.

[0103] A first object determination module, used for determining a plurality of first objects according to the object application request and a preset object length;

[0104] an adjustment information determination module, used to obtain the remaining storage space of the writable area, and determine the adjustment information of the object length according to the remaining storage space;

[0105] A second object determination module, configured to adjust the size and length of the first object according to the adjustment information to obtain a second object;

[0106] The writable area allocation module is used to pre-allocate capacity space according to the length of the second object, allocate a writable area according to the pre-allocated capacity space, and obtain write information corresponding to the file to be written.

[0107] In some embodiments, the adjustment information determination module includes: a remaining storage space determination module and a first adjustment information acquisition module.

[0108] A remaining storage space determination module, used to obtain the remaining storage space of the writable area;

[0109] The first adjustment information acquisition module is used to determine that the adjustment information of the object length is to reduce the object length when the remaining storage space of the writable area is less than the object length; wherein the reduced object length is not less than the data size in the object.

[0110] In some embodiments, the adjustment information determination module further includes: a second adjustment information acquisition module.

[0111] The second adjustment information acquisition module is used to reduce the object length to the data size in the object.

[0112] In some of the embodiments, the writable area allocation module includes: a writable area acquisition module, a capacity space pre-allocation module, a writing module and a writing information determination module.

[0113] A writable area acquisition module, used to acquire at least one writable area;

[0114] A capacity space pre-allocation module, used for acquiring a corresponding pre-allocated capacity space according to the length of each of the second objects;

[0115] A writing module, used for writing the corresponding second objects into the remaining storage space of the writable area in sequence according to the pre-allocated capacity space;

[0116] The writing information determining module is used to determine the writing information corresponding to the file to be written according to the writable area to be written; the index information of the writable area and the length of each second object in the writable area.

[0117] In some of the embodiments, the distributed object storage device further includes: a remaining storage space acquisition unit.

[0118] The remaining storage space acquisition unit is used to obtain the remaining storage space of the writable area uploaded by the storage node; the remaining storage space is calculated by the storage node based on the pre-allocated capacity space of at least one second object in the writable area after the file to be written is written to the storage node.

[0119] In some of the embodiments, the distributed object storage device further includes: a length acquisition unit and a space correction unit.

[0120] A length obtaining unit, used for obtaining the length of each second object actually written into the writable area;

[0121] The space correction unit is used to correct the pre-allocated capacity space to the length of the second object actually written into the writable area when the length of the second object actually written into the writable area is smaller than the corresponding pre-allocated capacity space.

[0122] In some of the embodiments, the write information further includes the number of objects in the writable area, and the distributed object storage device further includes: a persistence unit and a deletion unit.

[0123] A persistence unit, used for persistently storing the written information;

[0124] The deleting unit is used to delete the corresponding writable area when the number of objects in the writable area is zero.

[0125] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0126] In addition, combined Figure 1 The distributed object storage method described in the embodiment of the present application can be implemented by a metadata server. Figure 8 Schematic diagram of the hardware structure of the metadata server according to an embodiment of the present application.

[0127] The metadata server may include a processor 81 and a memory 82 storing computer program instructions.

[0128] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0129] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0130] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0131] The processor 81 implements any one of the distributed object storage methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0132] In some embodiments, the metadata server may further include a communication interface 83 and a bus 80. Figure 8 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0133] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0134] The bus 80 includes hardware, software or both, and couples the components of the metadata server to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0135] The metadata server can execute the distributed object storage method in the embodiment of the present application based on the obtained program instructions, thereby realizing the combination Figure 4 Describes a distributed object storage method.

[0136] like Fig. 9 As shown, this embodiment also provides a distributed object storage system, including: a client 103, at least one storage node 102 and a metadata server 101; wherein the metadata server 101 is communicatively connected with each storage node 102 and the client 103.

[0137] In this embodiment, the client 103 is used to initiate an object application request to the metadata server 101, and write the file to be written to the corresponding storage node 102 according to the write information returned by the metadata server 101. The storage node 102 is used to store the file to be written by the client. Specifically, the client sends a write request to the metadata server 101, and when the storage node 102 receives the write request, it writes the file to be written to the corresponding writable area. After the write is completed, the write success information is returned. The metadata server 101 is used to execute the above-mentioned distributed object storage method, which is not described in detail in this application.

[0138] In some of the embodiments, the storage node 102 is further used to: calculate the remaining storage space of the corresponding writable area based on the pre-allocated capacity space of the second object in each writable area, and upload the remaining storage space of the writable area to the metadata server 101. In this embodiment, after receiving the write request sent by the client 103, the storage node 102 searches for the corresponding Zone according to the write request, and verifies whether the remaining size of the Zone meets the object length. If the verification passes, the file to be stored is appended to the Zone, the metadata is recorded, and the remaining storage space of the writable area is uploaded to the metadata server 101.

[0139] In addition, in combination with the distributed object storage method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the distributed object storage methods in the above embodiment is implemented.

[0140] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A distributed object storage method, characterized in that: include: When receiving an object application request initiated by a client, obtaining a writable area of ​​an SMR disk in a distributed storage system; wherein the object application request includes the size of a file to be written, and the SMR disk is divided into a plurality of writable areas; Determine the writing information according to the object application request and the remaining storage space of the writable area; the writing information includes the length of each object obtained by segmenting the file to be written; Returning the write information to the client, so that the client writes the file to be written into the corresponding storage node according to the write information; Determining the writing information according to the object application request and the remaining storage space of the writable area includes: Determining a plurality of first objects according to the object application request and a preset object length; Acquire the remaining storage space of the writable area, and determine the adjustment information of the object length according to the remaining storage space; Adjust the size and length of the first object according to the adjustment information to obtain a second object; pre-allocate capacity space according to the length of the second object, allocate a writable area according to the pre-allocated capacity space, and obtain write information corresponding to the file to be written; Obtaining the remaining storage space of the writable area and determining the adjustment information of the object length based on the remaining storage space includes: obtaining the remaining storage space of the writable area; when the remaining storage space of the writable area is less than the object length, determining the adjustment information of the object length to reduce the object length; wherein the reduced object length is not less than the data size in the object.

2. The distributed object storage method according to claim 1, characterized in that: When the remaining storage space of the writable area is less than the object length, the adjustment information for determining the object length is: The object length is reduced to the size of the data within the object.

3. The distributed object storage method according to claim 1, characterized in that: Pre-allocating capacity space according to the length of the second object, allocating a writable area according to the pre-allocated capacity space, and obtaining writing information corresponding to the file to be written includes: Get at least one writable area; Acquire the corresponding pre-allocated capacity space according to the length of each of the second objects; Writing the corresponding second objects into the remaining storage space of the writable area in sequence according to the pre-allocated capacity space; According to the writable area to be written, writing information corresponding to the file to be written is determined; index information of the writable area and the length of each second object in the writable area.

4. The distributed object storage method according to claim 1, characterized in that: After returning the write information to the client so that the client writes the to-be-written file into the corresponding storage node according to the write information, the method further includes: The remaining storage space of the writable area uploaded by the storage node is obtained; the remaining storage space is calculated by the storage node based on the pre-allocated capacity space of at least one second object in the writable area after the file to be written is written into the storage node.

5. The distributed object storage method according to claim 1, characterized in that: After returning the write information to the client so that the client writes the to-be-written file into the corresponding storage node according to the write information, the method further includes: Get the length of each second object actually written into the writable area; When the length of the second object actually written into the writable area is smaller than the corresponding pre-allocated capacity space, the pre-allocated capacity space is corrected to be the length of the second object actually written into the writable area.

6. The distributed object storage method according to claim 1, characterized in that: The write information also includes the number of objects in the writable area, and the method further includes: Persistently storing the written information; When the number of objects in the writable area is zero, the corresponding writable area is deleted.

7. A distributed object storage device, characterized in that: include: A writable area acquisition unit, configured to acquire a writable area of ​​an SMR disk in a distributed storage system when receiving an object application request initiated by a client; wherein the object application request includes the size of a file to be written, and the SMR disk is divided into a plurality of writable areas; A writing information determining unit, configured to determine writing information according to the object application request and the remaining storage space of the writable area; the writing information includes the length of each object obtained by segmenting the file to be written; A storage unit, used for returning the write information to the client, so that the client writes the file to be written into a corresponding storage node according to the write information; Determining the writing information according to the object application request and the remaining storage space of the writable area includes: Determining a plurality of first objects according to the object application request and a preset object length; Acquire the remaining storage space of the writable area, and determine the adjustment information of the object length according to the remaining storage space; Adjust the size and length of the first object according to the adjustment information to obtain a second object; pre-allocate capacity space according to the length of the second object, allocate a writable area according to the pre-allocated capacity space, and obtain write information corresponding to the file to be written; Obtaining the remaining storage space of the writable area and determining the adjustment information of the object length based on the remaining storage space includes: obtaining the remaining storage space of the writable area; when the remaining storage space of the writable area is less than the object length, determining the adjustment information of the object length to reduce the object length; wherein the reduced object length is not less than the data size in the object.

8. A metadata server, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the distributed object storage method according to any one of claims 1 to 6.

9. A distributed object storage system, characterized in that: include: A client, at least one storage node and a metadata server; wherein the metadata server is in communication connection with each storage node and the client; The client is used to initiate an object application request to the metadata server, and write the file to be written into the corresponding storage node according to the writing information returned by the metadata server; The storage node is used to store the files to be written by the client; The metadata server is used to execute the distributed object storage method according to any one of claims 1 to 6.

10. The distributed object storage system according to claim 9, characterized in that: The storage node is also used for: The remaining storage space of the corresponding writable area is calculated based on the pre-allocated capacity space of the second object in each writable area, and the remaining storage space of the writable area is uploaded to the metadata server.

Images