A data management method and apparatus

Abstract

The specification provides a data management method and device, and relates to the technical field of communication. A data management method comprises: obtaining information of a hole formed in a Ceph and a space occupancy rate of an object space in the Ceph, wherein the information of the hole at least comprises a correspondence relationship among a hole object, a hole range and a hole state; and if it is determined that the space occupancy rate of the object space in the Ceph is greater than a first threshold value and to-be-written data falls into the hole range of the hole object, the to-be-written data is written into a storage space of an object device corresponding to the hole object, and the information of the hole is updated. In this way, the data storage efficiency in the Ceph can be improved.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of communication, and particularly relates to a data management method and device. BACKGROUND

[0002] With the development of network technology, people's demand for network storage also gradually increases, such as storage efficiency and storage speed. Ceph, as a distributed storage system, provides a software-defined, unified storage solution for block storage, object storage and file storage, so that Ceph is widely used in the field of data storage.

[0003] Data writing to Ceph and data reading from Ceph are realized through the interaction between the block device and the object device. After the data reaches the block device, it will be divided into multiple objects of a preset space size according to the demand, and each object has a certain object space. When data needs to be written to an object, the corresponding storage space needs to be applied to the object device (which can be understood as a hard disk) for storage.

[0004] When data is written to an object, there are two IO (Input Output) processing methods, namely sequential IO and random IO. In the processing method of random IO, data is scattered and written into the object space. If there is an unoccupied part in the object space, but the head and tail of the object space are written, the storage space needs to be applied to the object device for storage. At this time, these unoccupied parts are called holes.

[0005] In Ceph, the occupancy of the object device is monitored according to the applied storage space, but due to the existence of holes, Ceph may consider that the object device has been written full, but in fact there is a large amount of storage space that is not effectively occupied, thereby causing waste of storage space. SUMMARY

[0006] In order to overcome the problems in the related art, the present specification provides a data management method and device.

[0007] In combination with the first aspect of the embodiments of the present specification, the present application provides a data management method, comprising:

[0008] Obtaining hole information formed in Ceph and space occupancy of object space in Ceph, wherein the hole information at least includes the correspondence relationship of hole object, hole range and hole state;

[0009] If it is determined that the space occupancy of the object space in Ceph is greater than a first threshold and the to-be-written data falls into the hole range of the hole object, the to-be-written data is written to the storage space of the object device corresponding to the hole object, and the hole information is updated.

[0010] Optionally, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0011] If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the first hole which is in an idle state, then the data to be written is written to the storage space of the object device corresponding to the first hole.

[0012] Remove the first hole from the hole information.

[0013] Optionally, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0014] If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the second hole which is in a non-idle state, then the data currently stored in the second space is migrated to the third hole which is in an idle state, and the data to be written is written to the storage space of the object device corresponding to the second hole.

[0015] Remove the second hole from the hole information and set the third hole to a non-idle state.

[0016] Optionally, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0017] If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the fourth hole where no hole object has been created, then a new hole object is partitioned from the object space and the data to be written is written into the new hole object.

[0018] Record the hole information corresponding to the new hole object.

[0019] Optionally, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0020] If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the fifth hole which is in a non-idle state, then the data to be written will overwrite the data in the storage space of the object device corresponding to the fifth hole.

[0021] If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written to the hole object falls into the sixth hole which is in an idle state, then the data to be written is written to the storage space of the object device corresponding to the sixth hole, and the sixth hole is changed to a non-idle state.

[0022] Optionally, the method further includes:

[0023] If it is determined from the read request that data is to be read from the hole object, then the first data is read from the object device corresponding to the hole object and the first data is sent.

[0024] If it is determined from the read request that data should be read from the original object, then the second data is read from the object device corresponding to the original object. The data corresponding to the void object contained in the second data is eliminated according to the void information to form the third data, and the third data is sent.

[0025] Optionally, the method further includes:

[0026] If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the empty space in the object space is less than the second threshold, then the writing of the data to be written will be blocked.

[0027] In conjunction with a second aspect of the embodiments described herein, this application provides a data management apparatus, comprising:

[0028] The acquisition unit is used to acquire hole information formed in Ceph and space occupancy rate of object space in Ceph. The hole information includes at least the correspondence between hole objects, hole ranges, and hole states.

[0029] The write unit is used to write the data to be written to the storage space of the object device corresponding to the hole object and update the hole information if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the hole range of the hole object.

[0030] Optional, the write unit includes:

[0031] The first writing module is used to write the data to be written to the storage space of the object device corresponding to the first hole if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the first hole which is in an idle state.

[0032] The first update module is used to remove the first hole from the hole information.

[0033] Optional, the write unit includes:

[0034] The second writing module is used to migrate the currently stored data in the second space to the third space in the idle state if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the second hole in the non-idle state, and write the data to be written to the storage space of the object device corresponding to the second hole.

[0035] The second update module is used to remove the second hole from the hole information and set the third hole to a non-idle state.

[0036] Optional, the write unit includes:

[0037] The third writing module is used to divide a new hole object from the object space and write the data to be written into the new hole object if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the fourth hole where no hole object has been created.

[0038] The third update module is used to record the hole information corresponding to the new hole object.

[0039] Optional, the write unit includes:

[0040] The fourth write module is used to overwrite the data in the storage space of the object device corresponding to the fifth hole if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the fifth hole which is in a non-idle state; if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the sixth hole which is in an idle state, then the data to be written is written into the storage space of the object device corresponding to the sixth hole.

[0041] The fourth update module is used to remove the sixth hole from the hole information.

[0042] Optionally, the device may also include:

[0043] The reading unit is configured to, if it is determined from the reading request that data be read from the hole object, read first data from the object device corresponding to the hole object and send the first data; if it is determined from the reading request that data be read from the original object, read second data from the object device corresponding to the original object, eliminate the data corresponding to the hole object contained in the second data according to the hole information to form third data, and send the third data.

[0044] Optionally, the device may also include:

[0045] The blocking unit is used to block the writing of data if it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the hole space in the object device is less than a second threshold.

[0046] In conjunction with a third aspect of the embodiments of this specification, this application provides an electronic device including a transceiver, a processor, and a machine-readable storage medium storing machine-executable instructions executable by the processor, which in turn causes the processor to perform the method steps of any of the preceding embodiments.

[0047] In conjunction with a fourth aspect of the embodiments of this specification, this application provides a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method steps of any of the above-described embodiments.

[0048] The technical solutions provided by the embodiments in this specification may include the following beneficial effects:

[0049] In the embodiments described in this specification, when the space occupancy rate of the object space in Ceph exceeds a first threshold, causing the original object to be unable to be written, write requests to the object space continue to be received. Based on the hole information recorded in Ceph, it is determined whether there is any data to be written within the hole range of the hole object. If so, the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated. When the space occupancy rate of the requested storage space of the object device in Ceph reaches a point where it is impossible to receive the data to be written, the space of the hole is used to continue storing data, thereby improving the storage efficiency of the object device.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0052] Figure 1 This is a flowchart of a data management method involved in this application;

[0053] Figure 2 This is a schematic diagram of Ceph in a data management method involved in this application;

[0054] Figure 3 This is a schematic diagram of a single data write operation in a data management method involved in this application;

[0055] Figure 4 This is a schematic diagram illustrating the changes in object state in a data management method according to an embodiment of this application;

[0056] Figure 5 This is a schematic diagram illustrating the change of object state in a data management method according to another embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a data management device involved in this application;

[0058] Figure 7 This is a schematic diagram of the structure of a network element involved in this application. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification.

[0060] This application provides a data management method, such as Figure 1 As shown, it includes:

[0061] S100: Obtain information on holes formed in Ceph and the space occupancy rate of objects in Ceph.

[0062] This method can be applied to electronic devices in Ceph, which can be understood as physical servers, virtual servers, or hosts with similar functions to servers. The following description uses electronic devices as an example of servers, but this does not constitute a limitation on electronic devices.

[0063] Ceph (also known as a Ceph cluster) includes clients, monitors, and OSDs (Object Storage Devices). Ceph can contain multiple monitors and OSDs. An OSD can be understood as a hard drive to store and retrieve data.

[0064] Multiple resource pools can be created on the client side, such as Figure 2As shown, an RBD (Rados Block Device) is mounted in the resource pool, and multiple PGs (Placement Groups) are configured in the resource pool. PGs can be mapped to several OSDs. When an RBD receives data to be written, it can divide it into blocks according to a preset size, such as 4MB (Mega Byte). These blocks can be called objects.

[0065] In the resource pool, after the RBD receives the data to be written and forms an object, it is mapped to the PG. The PG can record the object space of the managed objects, such as the storage space table. Through the PG, the data load in the object can be distributed to OSDs for storage. Ceph determines the space occupancy of OSDs by analyzing the storage space tables in the resource pool. Furthermore, since data writing to OSDs can be implemented based on load sharing, the space occupancy of one OSD can reflect the overall space occupancy of the OSDs mapped in this resource pool.

[0066] The information recorded in the storage space table is updated based on the object's request for storage space from the OSD. In other words, when writing via random I / O, the object requests a large amount of storage space from the OSD. Due to the existence of gaps, a large amount of storage space in the requested storage space may still not be actually written to, thus resulting in a waste of storage space in the OSD.

[0067] In order to record these wasted storage spaces, void information can also be recorded in the resource pool. This void information can include void objects, void ranges, and void states.

[0068] Among them, the original object refers to the object formed when it is able to request storage space from the OSD, and the void object refers to the object formed from the object space in the original object;

[0069] The hole range refers to the address range corresponding to each hole object;

[0070] The void state is used to indicate whether a void object stores data.

[0071] Within the hole information, a corresponding relationship is established between the hole object, the hole extent, and the hole state. Specifically, at least three object properties (Object map, hereinafter referred to as Omap properties) can be added to the object, namely:

[0072] object_hollow_size: Used to record the total size of the hollow space contained in the object space of a given object;

[0073] object_hollow_status: Used to record the range of the hollow space contained in the original object and the status of the hollow space within that range. It can be determined based on the recorded identifier. For example, when marked as "0", it means that the hollow space within that range is not occupied, and when marked as "1", it means that the hollow space within that range is occupied.

[0074] object_space_map: Used to record the object name and the extent of the hole objects generated from the original object.

[0075] Based on the three additional omap attributes mentioned above, the hole space, hole state, and correspondence between the hole and the hole object in the original object can be determined.

[0076] In addition, PG also maintains a hole table, which records the original object and the size of the hole space contained within that original object. This hole table allows determination of the size of the hole space within the original objects contained in a resource pool. When data needs to be written to a hole object, the load balancing of data writing can be performed based on this hole table.

[0077] During the data writing process, changes in the object space or hole space may cause changes in the three newly added Omap attributes mentioned above, as well as changes in the hole space in the hole table. Since the storage space table records the object space occupied by the original object that has applied for storage space in the OSD, using holes for data writing generally will not affect the changes in the storage space table.

[0078] S101. If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated.

[0079] The object space of the original object is recorded in the storage space table of the PG. Based on the remaining object space and the total object space, the space occupancy rate of the object space can be calculated. When the space occupancy rate is greater than a first threshold (e.g., 90%, set according to requirements, no limit is imposed), it can be considered that the original object has requested a large amount of storage space from the OSD. At this time, in the original Ceph, data writing is generally blocked to avoid problems such as data overflow of the OSD. This state can be called the full state.

[0080] In this application, if it is determined that the space occupancy rate is greater than the first threshold (i.e., a resource pool is full), the write requests sent by the client can still be received, but different processing can be performed according to the objects pointed to in the write requests. Moreover, a completely full state can be added, and when the completely full state is reached, the writing of data will be blocked.

[0081] When the space occupancy rate of the object space is not greater than the first threshold, after receiving a write request, Ceph can parse it to obtain a file. As Figure 3 shown, RBD can split the file to form several original objects and write them into the object space of the original objects. These original objects will be uniquely named in the form of <RBD number. object number>.

[0082] After the occupancy rate of the object space is greater than the first threshold, Ceph continues to parse the received write requests. After obtaining a new file and splitting it in RBD, it can determine which objects have hole spaces based on the hole table, and create hole objects through these hole spaces. These hole objects will be uniquely named in the form of <RBD number. original object number. hole object number>.

[0083] Falling into the hole space of a hole object means that the data to be written will be written into the hole space occupied by the hole object, but it does not mean that the data to be written directly points to the hole object. It is possible that the data to be written directly pointing to the hole object falls into the hole space of the hole object, or it is also possible that the data to be written pointing to the original object falls into the hole space of the hole object.

[0084] After the data to be written is written into the hole space, whether the data to be written points to an original object or a hole object, it will cause a change in the hole information. Therefore, after the data to be written is finally written into the OSD, Ceph also needs to update the hole information it maintains, so that the hole information can reflect the situation of the original objects and hole objects in Ceph in real time.

[0085] According to the different objects pointed to by the data to be written, the following different processing methods can be carried out.

[0086] Method 1:

[0087] Optionally, in step S101, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the hole range of the hole object, then write the data to be written into the storage space of the object device corresponding to the hole object and update the hole information, including:

[0088] S101A. If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the first hole which is in an idle state, then the data to be written is written to the storage space of the object device corresponding to the first hole.

[0089] The write requests received by Ceph are for files that have already been written to; that is, the data to be written, resulting from the RBD splitting of the file, will be directed to the original object. For example, if... Figure 4 As shown, the example of writing data to be written to an original object in the resource pool is used for illustration.

[0090] At the RBD, based on a size of 4MB, the file is divided into 5 blocks, each pointing to one of the 5 original objects, labeled as objects 1-5. During the last write to object 3, 3 empty objects were created within it: object 30, object 31, and object 32. Therefore, the omap attribute recorded for object 3 can be represented as follows:

[0091] object_hollow_size: X1 bits, where X1 is the sum of the hole spaces of objects 30, 31, and 32;

[0092] object_hollow_status: {"address1~address2":0","address3~address4":1","address5~address6":0};

[0093] object_space_map: {"object30": "address1~address2", "object31": "address3~address4", "object32": "address5~address6"}.

[0094] Based on the above records, it can be concluded that three empty objects were created under the original object numbered 3: object 30, object 31, and object 32. These three empty objects correspond to the empty spaces in object 3. In other words, each empty object represents an empty space and occupies a segment of empty space.

[0095] Since the data to be written that points to the original object has a higher priority, when it is determined that the data to be written points to the original object and needs to be written to each hole of the original object, it is necessary to determine whether the hole is in an idle state.

[0096] In Method 1, after determining that the data to be written falls into the hollow space "address 1~address 2" (which can be called the first hollow) used by the hollow object, and if the "address 1~address 2" is determined to be in an idle state according to the record of object_hollow_status, then it is determined that it can be written directly.

[0097] After being written to the empty space, the data in the object is written to the OSD through the mapping between the storage space table recorded in the PG and the OSD.

[0098] S101B, Remove the first void from the void information.

[0099] When the data to be written is written into the first hole, it indicates that the data for the original object occupies the space of the first hole. At this time, it can be said that in object 3, object 30, as the hole object, will occupy the space for writing the original object, and the hole will not exist. Therefore, it is necessary to remove the information related to the first hole from the hole information.

[0100] The aforementioned hole information will then be updated as follows:

[0101] object_hollow_size: X2 bits, where X2 is the sum of the hole spaces of object 31 and object 32;

[0102] object_hollow_status: { “address 3~address 4”:1, “address 5~address 6”:0};

[0103] object_space_map: {"object31": "address3~address4", "object31": "address5~address6"}.

[0104] That is, it can be seen from the updated hole information that since the data to be written to the original object occupies object 30 (the first hole), which is a hole object, the hole occupied by the original object needs to be removed from the hole information.

[0105] Method 2:

[0106] Optionally, in step S101, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0107] S101C. If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the second hole which is in a non-idle state, then the data currently stored in the second space is migrated to the third hole which is in an idle state, and the data to be written is written to the storage space of the object device corresponding to the second hole.

[0108] In the context of Method 1, if the data to be written falls into the hollow space "address 3~address 4" (which can be called the second hollow) used by the hollow object, "address 3~address 4" is in a non-idle state as determined by the record of object_hollow_status. That is to say, it has been occupied by the data to be written pointing to the hollow object.

[0109] At this point, it is necessary to first migrate the data already written to the empty space to other empty spaces, that is, to restore the empty space. For example... Figure 5 As shown, the data in the second hole, which is not idle, is first migrated to the third hole, which is idle, and then the data to be written is written to the third hole.

[0110] After being written to the empty space, the data in the object is written to the OSD through the mapping between the storage space table recorded in the PG and the OSD.

[0111] S101D: Remove the second hole from the hole information and set the third hole to a non-idle state.

[0112] In Method 2, the empty space occupied by the empty object needs to be removed (i.e., the second empty space, object 31, "address 3~address 4"), and the data in the second empty space is written to the third empty space, that is, the data that originally pointed to the empty object is written to another empty space (i.e., the third empty space, object 32, "address 5~address 6").

[0113] The data to be written to the empty object is occupying object 31 (the second empty object), which is in a non-idle state. At this time, according to the priority order, when it is determined that a non-idle empty object needs to be written, it is necessary to ensure that the data pointing to the original object is written in the high priority order. Therefore, it is necessary to restore the object space of the already occupied empty object to an idle state, that is, to perform data migration.

[0114] After the migration, the data to be written is written to the second hole. Since the second hole is occupied by the original object, the hole object (i.e., object 31) needs to be removed from the hole information. Furthermore, since the data in object 31 has been migrated to the third hole (i.e., object 32), the third hole, which was originally in an idle state, needs to be placed into a non-idle state.

[0115] The hole information will be updated to:

[0116] object_hollow_size: X3 bits, where X3 is the sum of the hole spaces of object 30 and object 32;

[0117] object_hollow_status: {"address1~address2":0","address5~address6":1};

[0118] object_space_map: {"object30": "address1~address2", "object32": "address5~address6"}.

[0119] The above method can restore the object space of the original object occupied by the empty object, so as to ensure that the data pointing to the original object is reliably written and that the migrated data is not lost.

[0120] As can be seen from Method 1 and Method 2, for a hole, when the data to be written pointing to the hole object is written, the hole information is updated by changing the hole state. However, if the data to be written pointing to the original object is to be written into the hole, that is, the hole will be occupied by the original object, it can be understood that the hole occupied by the original object is no longer called a hole. Therefore, the hole information is updated by removing the hole object and the hole range.

[0121] Method 3:

[0122] Optionally, in step S101, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0123] S101E: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the fourth hole where no hole object has been created, then a new hole object is divided from the object space, and the data to be written is written into the new hole object.

[0124] If the data to be written needs to be written to a hole object, but a hole object has not yet been created in the corresponding original object, the required hole space size is determined based on the hole table generated in PG. For example, if the data to be written requires 8MB of hole space, and object 4 recorded in the hole table contains 16MB of storage space, several hole objects can be divided in object 4.

[0125] The hole object generated in object 4 is as follows:

[0126] object_hollow_size: X4 bits, where X4 is the sum of the hole spaces of objects 40, 41, 42, and 43;

[0127] object_hollow_status: {"address 7~address 8":0","address 9~address 10":0","address 11~address 12":0","address 13~address 14":0};

[0128] object_space_map: {"object40": "address7~address8", "object41": "address9~address10", "object42": "address11~address12", "object43": "address13~address14"}.

[0129] Assuming the hole range of objects 40 and 41 is 8MB, the data to be written can be written into objects 40 and 41 (i.e., the fourth hole), occupying "address 7~address 8" and "address 9~address 10". After writing into the hole space, the data in the object is written into the OSD through the mapping between the storage space table recorded in the PG and the OSD.

[0130] S101F: Record the hole information corresponding to the new hole object.

[0131] After the data is written to the OSD, the hole information corresponding to the new hole object is updated, as follows:

[0132] object_hollow_size: X5 bits, where X5 is the sum of the hole spaces of objects 40, 41, 42 and 43;

[0133] object_hollow_status: {"address 7~address 8":1","address 9~address 10":1","address 11~address 12":0","address 13~address 14":0};

[0134] object_space_map: {"object40": "address7~address8", "object41": "address9~address10", "object42": "address11~address12", "object43": "address13~address14"}.

[0135] It can be seen that the hole status of the hole range of objects 40 and 41 is marked as non-idle, that is, the hole occupied by this data writing.

[0136] Method 4:

[0137] Optionally, in step S101, if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the void range of the void object, then the data to be written is written to the storage space of the object device corresponding to the void object, and the void information is updated, including:

[0138] S101G: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the fifth hole which is in a non-idle state, then the data to be written will overwrite the data in the storage space of the object device corresponding to the fifth hole.

[0139] Assuming the hole object created in object 4 is as follows:

[0140] object_hollow_size: X5 bits, where X5 is the sum of the hole spaces of objects 40, 41, 42 and 43;

[0141] object_hollow_status: {"address 7~address 8":1","address 9~address 10":1","address 11~address 12":0","address 13~address 14":0};

[0142] object_space_map: {"object40": "address7~address8", "object41": "address9~address10", "object42": "address11~address12", "object43": "address13~address14"}.

[0143] The data to be written this time points to an already created void object, and this void object is in a non-idle state, for example, writing object 40 (i.e., the fifth void mentioned above). Since the data to be written directly points to an existing void object, although data already exists in the void object, the data to be written to the void object can directly overwrite the existing data in the void object, thereby ensuring that the data is updated in the void.

[0144] Since object 40 (i.e. the fifth hole) has been marked as non-idle, the updated hole information is consistent with the previous information after the data to be written is written.

[0145] S101H: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the sixth hole which is in an idle state, then the data to be written is written to the storage space of the object device corresponding to the sixth hole, and the sixth hole is changed to a non-idle state.

[0146] The data to be written this time points to an already created void object, and this void object is in an idle state. For example, write object 42 (i.e., the sixth void mentioned above). Since the data to be written directly points to an existing void object, which does not contain any data, the data to be written to the void object can be directly written into the void object.

[0147] Since object 42 (i.e., the sixth hole) is marked as idle, the updated hole information after the data to be written puts object 42 into a non-idle state.

[0148] As can be seen from method 4 above, when the data to be written points to a hole object, it means that the data to be written is written to the hole space through the hole object. Therefore, if the hole is recorded in the hole information and there is data in the hole object, it means that the new data to be written is written to the hole object, so that the new data can overwrite the original data.

[0149] It should be noted that methods 1 through 4 can be executed by the Ceph server. The specific execution method is determined by the different objects (including original objects and empty objects) that the data to be written points to and the different states of the storage space where the data to be written falls. In other words, methods 1 through 4 can be processed by the Ceph server separately or simultaneously.

[0150] To enable data reading from the void space using the aforementioned method, the method may optionally further include:

[0151] S102. If it is determined from the read request that data is to be read from the void object, then the first data is read from the object device corresponding to the void object and the first data is sent.

[0152] Reading data from a void object is similar to the writing process. Data is read from the void object pointed to in the read request, and the first piece of data read is returned to the device that sent the read request (such as the client).

[0153] S103. If it is determined from the read request that data is to be read from the original object, then the second data is read from the object device corresponding to the original object, the data corresponding to the void object contained in the second data is eliminated according to the void information to form the third data, and the third data is sent.

[0154] Upon receiving a read request, if it is determined that data is being read from the original object, it is necessary to consider whether the object space of the original object contains data written through a hole object. At this point, the three newly set Omap properties in the original object can be checked to determine whether the original object contains a hole object and which hole object is in a non-idle state (i.e., data has been written through the hole state).

[0155] The hole information is shown below:

[0156] object_hollow_size: X5 bits, where X5 is the sum of the hole spaces of objects 40, 41, 42 and 43;

[0157] object_hollow_status: {"address 7~address 8":1","address 9~address 10":1","address 11~address 12":0","address 13~address 14":0};

[0158] object_space_map: {"object40": "address7~address8", "object41": "address9~address10", "object42": "address11~address12", "object43": "address13~address14"}.

[0159] As can be seen from the above hole information, data is written in objects 40 and 41. Therefore, when reading data through the original objects, the data written through the hole objects needs to be cleared. This clearing can be done by masking the "addresses 7~8" and "addresses 9~10" corresponding to objects 40 and 41, thus restoring the data written in the object space of the original objects.

[0160] When the empty space has been written to a certain extent, it means that the object space in a Ceph resource pool is basically fully occupied, and data writing cannot be achieved even through the empty space. To avoid data overflow caused by Ceph continuing to receive and process write requests in this situation, the method may also include:

[0161] S104. If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the empty space in the object space is less than the second threshold, then the writing of the data to be written is blocked.

[0162] After the storage space occupancy rate is determined to be greater than the first threshold through the storage space table, Ceph enters the full state. In this application, Ceph in the full state can still receive write requests and write.

[0163] Since the storage space table is nearly full, writes to the original object will not create a new original object, nor will storage space be requested from the OSD based on the original object. Subsequent data writes will allocate space based on the hole table. When the hole space recorded in the hole table is less than the second threshold, it indicates that the hole space is also mostly occupied, and further data reception cannot be written. At this point, data writing needs to be blocked to prevent data overflow.

[0164] Correspondingly, this application provides a data management device, such as Figure 6 As shown, it includes:

[0165] The acquisition unit is used to acquire hole information formed in Ceph and space occupancy rate of object space in Ceph. The hole information includes at least the correspondence between hole objects, hole ranges, and hole states.

[0166] The write unit is used to write the data to be written to the storage space of the object device corresponding to the hole object and update the hole information if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the hole range of the hole object.

[0167] Optional, the write unit includes:

[0168] The first writing module is used to write the data to be written to the storage space of the object device corresponding to the first hole if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the first hole which is in an idle state.

[0169] The first update module is used to remove the first hole from the hole information.

[0170] Optional, the write unit includes:

[0171] The second writing module is used to migrate the currently stored data in the second space to the third space in the idle state if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the second hole in the non-idle state, and write the data to be written to the storage space of the object device corresponding to the second hole.

[0172] The second update module is used to remove the second hole from the hole information and set the third hole to a non-idle state.

[0173] Optional, the write unit includes:

[0174] The third writing module is used to divide a new hole object from the object space and write the data to be written into the new hole object if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the fourth hole where no hole object has been created.

[0175] The third update module is used to record the hole information corresponding to the new hole object.

[0176] Optional, the write unit includes:

[0177] The fourth write module is used to overwrite the data in the storage space of the object device corresponding to the fifth hole if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the fifth hole which is in a non-idle state; if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the sixth hole which is in an idle state, then the data to be written is written into the storage space of the object device corresponding to the sixth hole.

[0178] The fourth update module is used to remove the sixth hole from the hole information.

[0179] Optionally, the device may also include:

[0180] The reading unit is configured to, if it is determined from the reading request that data be read from the hole object, read first data from the object device corresponding to the hole object and send the first data; if it is determined from the reading request that data be read from the original object, read second data from the object device corresponding to the original object, eliminate the data corresponding to the hole object contained in the second data according to the hole information to form third data, and send the third data.

[0181] Optionally, the device may also include:

[0182] The blocking unit is used to block the writing of data if it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the hole space in the object device is less than a second threshold.

[0183] The above-mentioned device can be applied to electronic devices in Ceph. These electronic devices can be understood as physical servers, virtual servers, or hosts with similar functions to servers. The following description uses electronic devices as an example of servers, but this does not constitute a limitation on electronic devices.

[0184] Correspondingly, this application provides an electronic device, such as Figure 7 As shown, it includes a transceiver, a processor, and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor, which in turn causes the processor to implement the method steps of any of the above.

[0185] Correspondingly, this application provides a machine-readable storage medium storing machine-executable instructions, which, when called and executed by a processor, cause the processor to implement any of the above-mentioned method steps.

[0186] The technical solutions provided by the embodiments in this specification may include the following beneficial effects:

[0187] In the embodiments described in this specification, when the space occupancy rate of the object space in Ceph exceeds a first threshold, causing the original object to be unable to be written, write requests to the object space continue to be received. Based on the hole information recorded in Ceph, it is determined whether there is any data to be written within the hole range of the hole object. If so, the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated. When the space occupancy rate of the requested storage space of the object device in Ceph reaches a point where it is impossible to receive the data to be written, the space of the hole is used to continue storing data, thereby improving the storage efficiency of the object device.

[0188] It should be understood that this specification is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

[0189] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. A data management method, characterized in that, include: Obtain hole information and object space occupancy rate in the distributed storage system Ceph. The hole information includes at least the correspondence between hole objects, hole ranges, and hole states. A hole object refers to an object formed from the object space of the original object. The original object refers to an object formed when it is able to apply for storage space from the object storage device OSD. The hole range refers to the address range corresponding to each hole object. The hole state is used to indicate whether the hole object stores data. If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls within the hole range of the hole object, then the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated. The space occupancy rate being greater than the first threshold indicates that Ceph has entered a full state, and Ceph in a full state can receive write requests and write.

2. The method according to claim 1, characterized in that, If it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the data to be written falls within the hole range of the hole object, then the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated, including: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the first hole which is in an idle state, then the data to be written is written to the storage space of the object device corresponding to the first hole. Remove the first void from the void information.

3. The method according to claim 1, characterized in that, If it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the data to be written falls within the hole range of the hole object, then the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated, including: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the second hole which is in a non-idle state, then the data currently stored in the second hole is migrated to the third hole which is in an idle state, and the data to be written is written to the storage space of the object device corresponding to the second hole. Remove the second void from the void information and set the third void to a non-idle state.

4. The method according to claim 1, characterized in that, If it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the data to be written falls within the hole range of the hole object, then the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated, including: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the fourth hole where no hole object has been created, then a new hole object is divided from the object space and the data to be written is written into the new hole object. Record the hole information corresponding to the new hole object.

5. The method according to claim 1, characterized in that, If it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the data to be written falls within the hole range of the hole object, then the data to be written is written to the storage space of the object device corresponding to the hole object, and the hole information is updated, including: If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the fifth hole which is in a non-idle state, then the data to be written will overwrite the data in the storage space of the object device corresponding to the fifth hole. If it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the sixth hole which is in an idle state, then the data to be written is written into the storage space of the object device corresponding to the sixth hole, and the sixth hole is changed to a non-idle state.

6. The method according to claim 1, characterized in that, Also includes: If it is determined from the read request that data is to be read from the void object, then the first data is read from the object device corresponding to the void object and the first data is sent. If it is determined from the read request that data is to be read from the original object, then the second data is read from the object device corresponding to the original object, the data corresponding to the void object contained in the second data is eliminated according to the void information to form the third data, and the third data is sent.

7. The method according to claim 1, characterized in that, Also includes: If it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the empty space in the object space is less than a second threshold, then the writing of the data to be written is blocked. Here, the empty space being less than the second threshold means that after Ceph is in a full state, data cannot be written.

8. A data management device, characterized in that, include: The acquisition unit is used to acquire hole information formed in Ceph and the space occupancy rate of object space in Ceph. The hole information includes at least the correspondence between hole objects, hole ranges, and hole states. The hole object refers to an object formed from the object space of the original object. The original object refers to an object formed when it is able to apply for storage space from the object storage device OSD. The hole range refers to the address range corresponding to each hole object. The hole state is used to indicate whether the hole object stores data. The writing unit is configured to, if it is determined that the space occupancy rate of the object space in the Ceph is greater than a first threshold and the data to be written falls within the void range of the void object, write the data to be written to the storage space of the object device corresponding to the void object and update the void information. The space occupancy rate being greater than the first threshold indicates that the Ceph has entered a full state, and the Ceph in the full state can receive write requests and write.

9. The apparatus according to claim 8, characterized in that, The writing unit includes: The first writing module is used to write the data to be written to the storage space of the object device corresponding to the first hole if it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the data to be written pointing to the original object falls into the first hole which is in an idle state. The first update module is used to remove the first void from the void information.

10. The apparatus according to claim 8, characterized in that, The writing unit includes: The second writing module is used to migrate the currently stored data in the second hole to the third hole in the idle state if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the original object falls into the second hole in the non-idle state, and write the data to be written to the storage space of the object device corresponding to the second hole. The second update module is used to remove the second hole from the hole information and set the third hole to a non-idle state.

11. The apparatus according to claim 8, characterized in that, The writing unit includes: The third writing module is used to divide a new hole object from the object space and write the data to be written into the new hole object if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written falls into the fourth hole where no hole object has been created. The third update module is used to record the hole information corresponding to the new hole object.

12. The apparatus according to claim 8, characterized in that, The writing unit includes: The fourth write module is used to overwrite the data in the storage space of the object device corresponding to the fifth hole if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the fifth hole which is in a non-idle state; and to write the data to be written into the storage space of the object device corresponding to the sixth hole if it is determined that the space occupancy rate of the object space in Ceph is greater than the first threshold and the data to be written pointing to the hole object falls into the sixth hole which is in an idle state. The fourth update module is used to change the sixth void to a non-idle state.

13. The apparatus according to claim 8, characterized in that, Also includes: The reading unit is configured to, if it is determined from a reading request that data be read from a hole object, read first data from the object device corresponding to the hole object and send the first data; If it is determined from the read request that data is to be read from the original object, then the second data is read from the object device corresponding to the original object, the data corresponding to the void object contained in the second data is eliminated according to the void information to form the third data, and the third data is sent.

14. The apparatus according to claim 8, characterized in that, Also includes: A blocking unit is used to block the writing of data to be written if it is determined that the space occupancy rate of the object space in Ceph is greater than a first threshold and the hole space in the object device is less than a second threshold. The hole space being less than the second threshold means that data cannot be written after Ceph is in a full state.

15. An electronic device, characterized in that, The method includes a transceiver, a processor, and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor, which is prompted by the machine-executable instructions to perform the steps of the method according to any one of claims 1-7.

16. A machine-readable storage medium, characterized in that, The device stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the steps of the method described in any one of claims 1-7.

Images