Data writing method, apparatus and device

Abstract

The application provides a data writing method, device and equipment. The method comprises the following steps: determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue; in response to a writing instruction of service data, determining a first quantity of available objects in the available object queue; and performing a writing operation of the service data according to the first quantity and a second quantity of available objects required by the service data. By writing the service data into the object of the storage cluster, the abnormal data storage and the situation that the data cannot be accessed due to the limited quantity of redundant resources are avoided, the continuous writing of the service data is realized, and the reliability of the data storage is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data writing method, device and equipment. BACKGROUND

[0002] Data is prone to errors in the storage process, which adversely affects the stability of data storage.

[0003] In order to provide the stability of data storage, the current commonly used is a redundancy protection strategy, that is, by setting a certain amount of redundant resources for data storage, these redundant resources may include, for example, redundant copies, redundant storage media, redundant nodes, etc.

[0004] However, the number of redundant resources is limited, and when the data storage process, the number of offline resources exceeds the number of redundant resources, data storage will still be abnormal. SUMMARY

[0005] The present application provides a data writing method, device and equipment to improve the reliability of data storage.

[0006] In a first aspect, the present application provides a data writing method, comprising:

[0007] determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue;

[0008] in response to a write instruction of business data, determining a first number of available objects in the available object queue;

[0009] According to the first number and the second number of available objects required by the business data, performing the write operation of the business data.

[0010] In a possible implementation, the first available logical unit is determined from the storage cluster, and the object corresponding to the first available logical unit is added to the available object queue, comprising:

[0011] Starting from the first logical unit in the storage cluster, the state of each logical unit is determined in turn, the state being available or unavailable;

[0012] At least one object corresponding to the first available logical unit with an available state is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

[0013] In a possible implementation, the write operation of the business data is performed according to the first number and the second number of available objects required by the business data, comprising:

[0014] in a case where the first quantity is greater than or equal to the second quantity, writing the service data into available objects in the available object queue;

[0015] in a case where the first quantity is less than the second quantity, determining a second available logical unit from the storage cluster except the first available logical unit, determining a third quantity of available objects corresponding to the second available logical unit, and adding the third quantity of available objects into the available object queue to obtain an updated available object queue, the third quantity being equal to a difference between the first quantity and the second quantity, and writing the service data into available objects in the updated available object queue.

[0016] In a possible implementation, the method further includes:

[0017] in a case where the first quantity is less than the second quantity and a quantity of objects corresponding to the second available logical unit is less than the third quantity, determining a fourth quantity of target used objects with the earliest storage time from used objects in the storage cluster, the fourth quantity being a difference between the third quantity and the quantity of objects corresponding to the second available logical unit;

[0018] writing the service data into the first quantity of available objects, the objects corresponding to the second available logical unit, and the fourth quantity of target used objects.

[0019] In a possible implementation, before determining the first quantity of available objects in the available object queue, the method further includes:

[0020] in a case where the first available logical unit has an abnormal logical unit, determining at least one third available logical unit from the storage cluster except the first available logical unit;

[0021] adding objects corresponding to the at least one third available logical unit into the available object queue, and deleting objects corresponding to the abnormal logical unit in the available object queue, a quantity of the objects corresponding to the at least one third available logical unit being greater than or equal to a quantity of the objects corresponding to the abnormal logical unit.

[0022] In a possible implementation, the method further includes:

[0023] adding the objects corresponding to the abnormal logical unit into a to-be-repaired queue, the objects in the to-be-repaired queue being unavailable;

[0024] In a case where a number of objects included in the available object queue is less than a preset number, the object successfully repaired in the to-be-repaired queue is added to the available object queue.

[0025] In a possible implementation, the method further includes:

[0026] In a case where a preset condition is met, a target object interval is determined in the storage cluster, and the target object interval includes a preset number of objects;

[0027] According to the target object interval, the available object queue is updated;

[0028] The preset condition includes at least one of the following:

[0029] A number of objects included in the available object queue is less than or equal to a preset threshold;

[0030] All objects included in the available object queue are in an unavailable state;

[0031] The objects in the available object queue are emptied.

[0032] In a second aspect, the present application provides a data writing device, including:

[0033] A processing module is configured to determine a first available logical unit from a storage cluster, and add an object corresponding to the first available logical unit to an available object queue;

[0034] A determining module is configured to determine a first number of available objects in the available object queue in response to a write instruction of business data;

[0035] A writing module is configured to perform a write operation of the business data according to the first number and a second number of available objects required by the business data.

[0036] In a possible implementation, the processing module is specifically configured to:

[0037] Starting from a first logical unit in the storage cluster, states of each logical unit are determined in sequence, and the state is available or unavailable;

[0038] At least one object corresponding to the first available logical unit with the available state is added to the available object queue until a number of objects in the available object queue is equal to a preset number.

[0039] In a possible implementation, the writing module is specifically configured to:

[0040] write the service data into the available objects of the available object queue in a case that the first quantity is greater than or equal to the second quantity;

[0041] determine a second available logical unit from the storage cluster in a case that the first quantity is less than the second quantity; determine a third quantity of available objects corresponding to the second available logical unit, and add the third quantity of available objects into the available object queue to obtain an updated available object queue, the third quantity being equal to a difference between the first quantity and the second quantity; and write the service data into the available objects of the updated available object queue.

[0042] In a possible implementation, the writing module is further configured to:

[0043] determine a fourth quantity of target used objects with the earliest storage time from the used objects of the storage cluster in a case that the first quantity is less than the second quantity, and a quantity of objects corresponding to the second available logical unit is less than the third quantity, the fourth quantity being a difference between the third quantity and the quantity of objects corresponding to the second available logical unit;

[0044] write the service data into the first quantity of available objects, the objects corresponding to the second available logical unit, and the fourth quantity of target used objects.

[0045] In a possible implementation, before determining the first quantity of available objects in the available object queue, the determining module is further configured to:

[0046] determine at least one third available logical unit from the storage cluster in a case that the first available logical unit has an abnormal logical unit;

[0047] add objects corresponding to the at least one third available logical unit into the available object queue, and delete objects corresponding to the abnormal logical unit in the available object queue, a quantity of the objects corresponding to the at least one third available logical unit being greater than or equal to a quantity of the objects corresponding to the abnormal logical unit.

[0048] In a possible implementation, the processing module is further configured to:

[0049] add the objects corresponding to the abnormal logical unit into a to-be-repaired queue, the objects in the to-be-repaired queue being unavailable;

[0050] In a case where a number of objects included in the available object queue is less than a preset number, the object successfully repaired in the to-be-repaired queue is added to the available object queue.

[0051] In a possible implementation, the processing module is further configured to:

[0052] In a case where a preset condition is met, a target object interval is determined in the storage cluster, and the target object interval includes a preset number of objects;

[0053] According to the target object interval, the available object queue is updated;

[0054] The preset condition includes at least one of the following:

[0055] A number of objects included in the available object queue is less than or equal to a preset threshold;

[0056] The objects included in the available object queue are all in an unavailable state;

[0057] The objects in the available object queue are emptied.

[0058] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the data writing method of any one of the first aspect.

[0059] In a fourth aspect, the present application provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data writing method of any one of the first aspect.

[0060] The data writing method, device and equipment provided by the present application first determine a first available logical unit from the storage cluster, and add the objects corresponding to the first available logical unit to the available object queue. After receiving the write instruction of the business data, in response to the write instruction of the business data, the first number of available objects in the available object queue is determined, and the write operation of the business data is performed according to the first number and the second number of available objects required by the business data. By writing the business data into the objects of the storage cluster, the abnormal data storage caused by the limited number of redundant resources is avoided, and the situation that the data cannot be accessed is avoided, the continuous writing of the business data is realized, and the reliability of the data storage is improved. Since the available objects are added in the available object queue in advance, the frequency of real-time requesting objects from the storage cluster in the process of writing the business data into the objects is also reduced. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by a person of ordinary skill in the art without creative effort.

[0062] Figure 1 An application scenario schematic diagram provided for an embodiment of the present application;

[0063] Figure 2 A flowchart of a data writing method provided for an embodiment of the present application;

[0064] Figure 3 A service data writing process schematic diagram provided for an embodiment of the present application;

[0065] Figure 4 A full coverage processing schematic diagram of data provided for an embodiment of the present application;

[0066] Figure 5 An exception processing schematic diagram of a logical unit provided for an embodiment of the present application;

[0067] Figure 6 A structure schematic diagram of a data writing device provided for an embodiment of the present application;

[0068] Figure 7 An entity structure schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely in combination with the accompanying drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0070] Firstly, the related concepts involved in the present application are simply introduced.

[0071] OSD: object-based storage device, object storage device, corresponding to the disk on the server. The OSD is a hardware-level storage medium, used for storing all data and object storage in the cluster. A disk or partition can be used as an OSD.

[0072] PG: placement group, refers to a collection of objects, one PG corresponds to one or more objects. At the logical level, objects are saved in PG, and PG and OSD have a certain mapping relationship.

[0073] With the development of science and technology, various types of data sources are continuously generated. Among these data, a part of the data can be discarded after use, but another part of the data needs to be stored for a long time for subsequent reading and use.

[0074] Taking video data as an example, the video data records rich information, and through the storage of the video data, when the related information in the video data needs to be obtained subsequently, it can be read from the storage medium in time, which is very convenient. Therefore, it is of great significance to ensure the stability of the storage device in the data storage business and have a safe redundancy protection mechanism.

[0075] Currently in the field of cloud storage, the main method to improve storage stability is to use redundancy protection strategies such as multiple copies, erasure redundancy, cluster disaster recovery, etc. The essence of the redundancy protection strategy is to set a certain amount of redundant resources to protect the data.

[0076] Multiple copies are a scheme from the perspective of data, which saves multiple copies of the same data on resources or storage devices. Since the same data is stored in multiple copies, even if part of the data is abnormal, other normal data copies can be normally retrieved from the resources or storage devices.

[0077] Erasure redundancy is a scheme from the perspective of storage media, which improves storage stability by creating an erasure strategy. When storing data, in addition to setting resources or storage devices for storing data, redundant resources or redundant storage devices are also set. When the resources or storage devices for storing data are abnormal, the data on the abnormal resources or storage devices is written to the redundant resources or redundant storage devices, thereby ensuring the continuity of the data service and normally playing back the historical data service.

[0078] Cluster disaster recovery is a scheme from the perspective of nodes in the cluster, which provides storage stability by setting redundant nodes. In the cluster, if some nodes are down, the redundant nodes can take over the work of the down nodes, including storing data in the redundant nodes, thereby ensuring the normal operation of the service.

[0079] However, whether it is to set redundant copies, redundant storage media or redundant nodes, the number is limited. When the number of offline resources exceeds the number of redundant resources, data storage will still be abnormal.

[0080] For example, for the multi-copy strategy, if the number of abnormal data exceeds the number of redundant data, the data cannot be obtained by reading the redundant data; for example, for the erasure redundancy strategy, if the number of failed resources or devices exceeds the number of redundant resources or redundant storage devices, the data stored in all failed resources or devices cannot be written into the corresponding redundant resources or redundant storage devices; for example, for the cluster disaster recovery strategy, if the number of down nodes exceeds the number of redundant nodes, the services of all down nodes cannot be taken over by the corresponding redundant nodes.

[0081] That is, through the analysis of the above resource protection strategies, when the redundancy protection strategy exceeds the set upper limit, the cluster state will be abnormal, and the historical object data and new service data will have the risk of being unable to access and write.

[0082] Based on this, the embodiment of the application provides a data writing method. In view of the fact that the current data protection strategy may cause data loss and new service data to be unable to be written, the real-time file system (RTFS) and the storage cluster management system are combined to ensure continuous reading and writing of services without interruption under the condition that the index storage pool is degraded or normal after the data storage pool redundancy strategy fails. The scheme of the embodiment of the application will be introduced below with reference to the accompanying drawings.

[0083] First, the application scenario of the embodiment of the application is introduced. Figure 1 The application scenario of the embodiment of the application is introduced.

[0084] Figure 1 An application scenario provided by the embodiment of the application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the application scenario includes an RTFS file system and a storage cluster. The RTFS file system belongs to a business layer application, and the storage cluster belongs to an underlying storage medium. The RTFS file system and the storage cluster can be deployed on the same device or different devices, and the embodiment of the application does not limit this.

[0085] The storage cluster includes a plurality of logical units, as shown in FIG. 2. Figure 1 As shown in FIG. 2, three logical units are exemplified, which are PG1, PG2 and PG3. Each logical unit maps one or more OSDs. Figure 1 As shown in FIG. 3, a plurality of OSDs are exemplified, including OSD1, OSD2,..., OSDn. Each OSD corresponds to one or more objects. Figure 1 As shown in FIG. 4, obj1, obj2,..., obj32 are exemplified, and so on. Therefore, each logical unit corresponds to one or more objects.

[0086] The RTFS file system and the storage cluster can interact through a reliable autonomic distributed object store (RADOS) interface. For example, the RTFS file system can issue a write request to the storage cluster through the RADOS interface, request an object from the storage cluster, write data into an object of the storage cluster, and the like, and the storage cluster returns a request result to the RTFS file system through the RADOS interface, and the like.

[0087] In the above embodiments, the RTFS file system and the storage cluster can interact through a reliable autonomic distributed object store (RADOS) interface. For example, the RTFS file system can issue a write request to the storage cluster through the RADOS interface, request an object from the storage cluster, write data into an object of the storage cluster, and the like, and the storage cluster returns a request result to the RTFS file system through the RADOS interface, and the like. Figure 1 An application scenario applicable to the present application is introduced, and based on the application scenario of the example, the scheme of the embodiments of the present application is introduced. Figure 1 An application scenario applicable to the present application is introduced, and based on the application scenario of the example, the scheme of the embodiments of the present application is introduced. Figure 2 An application scenario applicable to the present application is introduced, and based on the application scenario of the example, the scheme of the embodiments of the present application is introduced.

[0088] Figure 2 A flowchart of the data write method provided by the embodiments of the present application is shown in FIG. 1, which includes the following steps. Figure 2

[0089] S21, a first available logical unit is determined from the storage cluster, and an object corresponding to the first available logical unit is added to an available object queue.

[0090] The storage cluster includes at least one logical unit, and for any one logical unit, the logical unit can be an available logical unit or an unavailable logical unit. The available logical unit refers to a logical unit in a normal state, and the unavailable logical unit refers to a logical unit in an abnormal state. Each logical unit corresponds to one or more objects, and if the logical unit is an available logical unit, the object corresponding to the logical unit is an available logical object.

[0091] The objects in the storage cluster can be used for data writing, and in order to improve the reliability of data writing, an available object queue can be constructed. Specifically, first, a first available logical unit is determined from the logical units included in the storage cluster. The number of the first available logical units is one or more, and the first available logical units are a subset of the available logical units included in the storage cluster, which can be all the available logical units in the storage cluster or part of the available logical units in the storage cluster.

[0092] In the storage cluster, since the first available logical unit is a logical unit in a normal state, the object corresponding to the first available logical unit is also a current available object. After the first available logical unit is determined, the object corresponding to the first available logical unit is added to the available object queue, and the available object queue includes a plurality of current available objects.

[0093] ​Optionally, in the embodiment of the present application, the size of each object is the same, and the type of data to be written is also known, so the number of objects that can be included in the available object queue is also certain. Taking video data as an example, the size of the video data is generally not more than 2G, so 32 objects can be set in the available object queue, and the size of each object is 64MB.

[0094] For any first available logical unit, all objects corresponding to the first available logical unit can be added to the available object queue, or part of the objects corresponding to the first available logical unit can be added to the available object queue, and the embodiment of the present application does not limit this.

[0095] S22, in response to the write instruction of the service data, determining a first number of available objects in the available object queue.

[0096] After receiving the write instruction of the service data, it is necessary to determine the first number of available objects in the available object queue, and the second number of available objects required by the service data.

[0097] For any one of the available objects, the object initially added to the available object queue belongs to the available object, but the state of the object can change because the object can not be used immediately. For example, due to some reasons, the object is abnormal, so that it becomes unavailable, for example, there can be other parallel services that use the object, resulting in a decrease in the number of available objects in the available object queue, etc. Therefore, before writing the service data, it is necessary to determine the first number of available objects in the available object queue, and the first number is greater than or equal to 0.

[0098] S23, according to the first number and the second number of available objects required by the service data, performing the write operation of the service data.

[0099] The second number of available objects required by the service data refers to the number of objects required to write the service data into the object of the storage cluster. Since the size of each object is fixed, and the size of the service data can be known, the second number of available objects required by the service data can be determined according to the size of the service data and the size of the object, and the second number is greater than or equal to 1.

[0100] After determining the first quantity and the second quantity, the write operation of the business data can be performed according to the relationship between the first quantity and the second quantity. The first quantity can be greater than or equal to the second quantity, or can be less than the second quantity. If the first quantity is greater than or equal to the second quantity, it indicates that the available objects in the available object queue are sufficient for the write of the business data, and therefore the business data is directly written into the available objects in the available object queue; if the first quantity is less than the second quantity, it indicates that the available objects in the available object queue are insufficient for the write of the business data, and therefore additional objects need to be allocated from the storage cluster for the write of the business data.

[0101] The data write method provided by the embodiments of the present application first determines a first available logical unit from the storage cluster, and adds an object corresponding to the first available logical unit to an available object queue. After receiving a write instruction of business data, the first quantity of available objects in the available object queue is determined in response to the write instruction of the business data, and the write operation of the business data is performed according to the first quantity and the second quantity of available objects required by the business data. By writing the business data into the object of the storage cluster, the situation that the data storage is abnormal and the data cannot be accessed due to the limited number of redundant resources is avoided, the continuous write of the business data is realized, and the reliability of the data storage is improved. Since the available objects are added in the available object queue in advance, the frequency of real-time requesting objects from the storage cluster in the process of writing the business data into the object is also reduced.

[0102] On the basis of any of the above embodiments, the scheme provided by the embodiments of the present application will be further introduced in combination with the accompanying drawings.

[0103] Figure 3 The process diagram of the write of the business data provided by the embodiments of the present application is shown in FIG. 1, which includes a plurality of logical units, such as PG1, PG2 and PG3, in the storage cluster. Each PG maps a corresponding OSD, for example, PG1 maps OSD1 and OSD3, PG2 maps OSD2 and OSD4, and PG3 maps OSD5 and OSDn (part of the corresponding relationship is not exhausted). Figure 3

[0104] First, a first available logical unit needs to be determined from the storage cluster, and an object corresponding to the first available logical unit is added to an available object queue. The process will be introduced in combination with FIG. 2. Figure 3

[0105] In one possible implementation, starting from the first logical unit in the storage cluster, the state of each logical unit is determined in sequence, and the state is available or unavailable. Then, at least one object corresponding to the first available logical unit with the available state is added to the available object queue, until the number of objects in the available object queue is equal to a preset number.​​

[0106] by Figure 3 For example, assuming that PG1, PG2 and PG3 are the first logical unit, the second logical unit and the third logical unit respectively, first determine whether the status of PG1 is available. If the status of PG1 is available, PG1 is determined as a first available logical unit, and then the object corresponding to PG1 is added to the available object queue.

[0107] After adding the object corresponding to PG1 to the available object queue, it is determined whether the number of objects in the available object queue is equal to the preset number. If not, the state of the second logical unit, that is, the state of PG2, is determined to be available. If the state of PG2 is unavailable, the state of the third logical unit, that is, PG3, is determined to be available.

[0108] If the status of PG3 is available, PG3 is also determined as a first available logical unit, and then the object corresponding to PG3 is added to the available object queue. If the number of objects corresponding to PG3 is greater than the difference between the number of objects currently included in the available object queue and the preset number, some of the objects corresponding to PG3 can be selected to be added to the available object queue. For example, the preset number is 32, the available object queue currently includes 30 objects, and the number of objects corresponding to PG3 is 5, then 2 of the 5 objects corresponding to PG3 can be selected and added to the available object queue. When the number of objects included in the available object queue reaches the preset number, the addition of new objects to the available object queue can be suspended.

[0109] The number of objects included in the available object queue changes dynamically. After an available object is added to the available object queue, the number of available objects in the available object queue will gradually decrease because other services may need to use the available object.

[0110] Therefore, after receiving a business data write instruction and before writing the business data, it is necessary to determine the first number of available objects in the available object queue, and perform the business data write operation based on the first number and the second number of available objects required by the business data.

[0111] When the first number is greater than or equal to the second number, the business data is written into the available objects in the available object queue, that is, the second number of available objects are selected from the first number of available objects, and the business data is written into the second number of available objects.

[0112] In the case that the first quantity is less than the second quantity, since the available objects in the available object queue are insufficient to implement the write of the service data, a second available logical unit other than the first available logical unit needs to be determined from the storage cluster. Then, a third quantity of available objects corresponding to the second available logical unit is determined, and the third quantity of available objects is added to the available object queue to obtain an updated available object queue, and the third quantity is equal to the difference between the first quantity and the second quantity. After the third quantity of available objects is added to the available object queue, the quantity of available objects included in the updated available object queue is the sum of the first quantity and the third quantity, and the third quantity is equal to the difference between the first quantity and the second quantity, so the sum of the first quantity and the third quantity is equal to the second quantity, that is, the quantity of available objects included in the updated available object queue is greater than or equal to the second quantity. In this case, the second quantity of available objects is determined in the updated available object queue, and the service data is written into the available objects of the updated available object queue.

[0113] In the process of writing the service data, for the case that the first quantity is less than the second quantity, since the available objects included in the available object queue are insufficient to implement the write of the service data, additional request needs to be made to the storage cluster to allocate a third quantity of available objects to be added to the available object queue. However, in some cases, the quantity of objects corresponding to the second available logical unit is less than the third quantity, so that even if all the remaining available objects in the storage cluster are added to the available object queue, it is still insufficient to write the service data. For this insufficient available object quantity case, in order to avoid the loss of the write of the service data, the write of the service data is implemented through the full coverage processing of the data, which will be introduced below. Figure 4 The process is introduced.

[0114] Figure 4 The full coverage processing of the data provided by the embodiments of the present application is shown in the schematic diagram as Figure 4 The storage cluster includes a plurality of logical units, such as PG1, PG2 and PG3, and each PG includes corresponding objects.

[0115] In the case that the first quantity of available objects included in the available object queue is less than the second quantity of available objects required by the service data, and the quantity of objects corresponding to the second available logical unit is less than the third quantity, a fourth quantity of target used objects with the earliest storage time can be determined in the used objects of the storage cluster, and the fourth quantity is the difference between the third quantity and the quantity of objects corresponding to the second available logical unit.

[0116] In the case that the first quantity of available objects included in the available object queue is less than the second quantity of available objects required by the service data, and the quantity of objects corresponding to the second available logical unit is less than the third quantity, a fourth quantity of target used objects with the earliest storage time can be determined in the used objects of the storage cluster, and the fourth quantity is the difference between the third quantity and the quantity of objects corresponding to the second available logical unit. Figure 4For example, the first available logical unit includes PG1 and PG3, and the second available logical unit is PG2. The first number of available objects in the current available object queue is 22, and the second number of available objects required by the service data is 28, so 6 available objects need to be requested from the storage cluster, i.e., the third number is equal to 6. However, the number of objects corresponding to PG2 is 3, i.e. Figure 4 obj6, obj7 and obj8 in the example in Figure 4 , the third number is 6, and the number of objects corresponding to PG2 is 3, so the fourth number is equal to 3. The number of used objects included in the storage cluster can exceed the fourth number, and in this case, the used object with the earliest storage time is preferentially selected as the target used object. The storage time of an object refers to the time when the object is written with data, and the used object with the earliest storage time is the object that is written with data earliest.

[0117] After the fourth number of target used objects are determined, the service data is written into the first number of available objects, the objects corresponding to the second available logical unit, and the fourth number of target used objects. For example, the target used objects in Figure 4 include obj1, obj2 and obj3, the first number of available objects is 22 available objects included in the current available object queue, the objects corresponding to the second available logical unit are the objects corresponding to PG2 in the example in Figure 5 , i.e., obj6, obj7 and obj8, and the fourth number of target used objects include obj1, obj2 and obj3, which collectively constitute the objects used for writing the service data.

[0118] For the target used object, the original data written in the target used object is deleted to make the newly written service data uninterrupted. The used object with the earliest storage time is preferentially released to write the service data. However, in the case of an abnormal storage cluster, the RTFS file system cannot perceive the state of the abnormal object in the storage cluster, so the number of available objects recorded in the RTFS file system is sufficient, but the number of available objects in the storage cluster is insufficient, which can cause data write loss. Therefore, in the process of writing the service data, the full-coverage operation is only performed when the number of available objects in the available object queue is insufficient and sufficient available objects cannot be requested from the storage cluster, and the service data is written through the used object.

[0119] After the object corresponding to the first available logical unit is added to the available object queue, the first available logical unit may be in an abnormal state, and the object corresponding to the abnormal first available logical unit is also abnormal, and the abnormal object is unavailable, and thus related processing of the abnormality needs to be performed.

[0120] Specifically, before determining the first quantity of available objects in the available object queue, it is detected whether there is an abnormal logical unit in the first available logical unit.

[0121] In a case where it is detected that there is an abnormal logical unit in the first available logical unit, at least one third available logical unit other than the first available logical unit can be determined from the storage cluster, and then the object corresponding to the at least one third available logical unit is added to the available object queue, and the object corresponding to the abnormal logical unit in the available object queue is deleted. The quantity of the object corresponding to the at least one third available logical unit is greater than or equal to the quantity of the object corresponding to the abnormal logical unit.

[0122] The following will be described in combination with Figure 5 the process of abnormality processing.

[0123] Figure 5 The abnormality processing of the logical unit provided in the embodiments of the present application is shown in the schematic diagram as Figure 5 The storage cluster includes a plurality of logical units, such as PG1, PG2 and PG3, and each PG includes a corresponding object.

[0124] Before determining the first quantity of available objects in the available object queue, the abnormal logical unit can be determined in combination with the CRUSH algorithm, so that when the business data is written, the business data is not written into the object corresponding to the abnormal logical unit.

[0125] For example, in Figure 5 , PG1 is a first available logical unit, and the object corresponding to PG1 includes obj9 and obj10. In a case where it is detected that PG1 is an abnormal logical unit, at least one third available logical unit needs to be determined from the storage cluster, that is, PG2 in Figure 5 , and then the object obj6, obj7 and obj8 corresponding to PG2 are added to the available object queue, and obj9 and obj10 are removed from the available object queue.

[0126] Further, for the abnormal logical unit, the object corresponding to the abnormal logical unit can be added to the repair queue for repair, and the object in the repair queue is an unavailable object. For example, Figure 6 , the object obj9 and obj10 corresponding to PG1 are added to the repair queue for repair.

[0127] In a case where the number of objects included in the available object queue is less than the preset number, the object in the repair-to-be-performed queue that has been successfully repaired can be added to the available object queue. For example, after obj9 and obj10 are added to the repair-to-be-performed queue for repair, if the repair is successful, in a case where the number of the available object queue is less than the preset number, obj9 and obj10 are added to the available object queue in time for writing of the business data. By adding the object that has been successfully repaired to the available object queue in time, the frequency of requesting the available object from the storage cluster can be reduced, and the efficiency of data storage can be improved.

[0128] For the storage cluster, all the OSDs are storage data resources, but the storage cluster cannot judge the validity of the object data. Therefore, the business layer can issue a corresponding structure rule, and the storage cluster returns a corresponding target object interval according to the corresponding structure rule, so that the business layer updates the available object queue.

[0129] Specifically, in a case where a preset condition is met, a target object interval is determined in the storage cluster, and the available object queue is updated according to the target object interval, wherein the target object interval includes a preset number of objects, and the preset condition includes at least one of the following:

[0130] Condition one: the number of objects included in the available object queue is less than or equal to a preset threshold.

[0131] When the available object queue is initially constructed, it includes a preset number of available objects, but as the business data is written, the number of available objects included in the available object queue gradually decreases. In a case where the number of available objects included in the available object queue is less than or equal to a preset threshold, for example, the available objects in the available object queue are used up, or the number of available objects is less than 5, the available object queue can be updated.

[0132] Condition two: all the objects included in the available object queue are in an unavailable state.

[0133] When an object is initially added to the available object queue, the object is available, but due to some reasons, the object is in an abnormal state, and the state of the object becomes an unavailable state. When all the objects included in the available object queue are in an unavailable state, the available object queue needs to be updated.

[0134] Condition three: the objects in the available object queue are emptied.

[0135] In a case where the business server system is restarted, the objects in the available object queue are all emptied, and the available object queue does not include any object. In this case, the available object queue needs to be updated.

[0136] Taking testObj_1~testObj_32 as an example, the target object region includes 32 objects. The service layer issues a structure rule to the storage cluster, and the structure rule is as follows:

[0137] {

[0138] ‘baseKey’:‘testObj’;

[0139] ‘obj_range’:0,32;

[0140] ‘map_used’:4032(00111111000000)

[0141] ‘policy’:‘free’

[0142] …

[0143] }

[0144] After receiving the structure rule, the storage cluster obtains the objects in the target object region testObj_1~testObj_32 and returns them to the service layer, so that the service layer updates the available object queue.

[0145] In summary, the data writing method provided by the embodiment of the present application reduces the frequency of the service layer requesting available objects from the storage cluster by requesting available objects in the storage cluster in advance and adding them to the available object queue. The performance load of the device is reduced, the data storage capacity is improved, the reliability of data storage is further ensured because the objects added to the available object queue are available objects, and continuous writing of business data is achieved.

[0146] The data writing device provided by the present application is described below. The data writing device described below can be referred to in conjunction with the data writing method described above.

[0147] Figure 6 The structure diagram of the data writing device provided by the embodiment of the present application is shown in FIG. 1, which includes: Figure 7

[0148] The processing module 61 is configured to determine a first available logical unit from the storage cluster and add an object corresponding to the first available logical unit to the available object queue.

[0149] The determining module 62 is configured to determine a first number of available objects in the available object queue in response to a writing instruction of business data.

[0150] ​The writing module 63 is configured to perform a writing operation of the service data according to the first quantity and a second quantity of available objects required by the service data.

[0151] In a possible implementation, the processing module 61 is specifically configured to:

[0152] Determine the state of each logical unit in sequence from the first logical unit in the storage cluster, the state being available or unavailable;

[0153] Add at least one object corresponding to the first available logical unit with the available state to the available object queue until the quantity of objects in the available object queue is equal to a preset quantity.

[0154] In a possible implementation, the writing module 63 is specifically configured to:

[0155] In a case where the first quantity is greater than or equal to the second quantity, write the service data into available objects in the available object queue;

[0156] In a case where the first quantity is less than the second quantity, determine a second available logical unit from the storage cluster except the first available logical unit, determine a third quantity of available objects corresponding to the second available logical unit, and add the third quantity of available objects to the available object queue to obtain an updated available object queue, the third quantity being equal to a difference between the first quantity and the second quantity; and write the service data into available objects in the updated available object queue.

[0157] In a possible implementation, the writing module 63 is further configured to:

[0158] In a case where the first quantity is less than the second quantity, and the quantity of objects corresponding to the second available logical unit is less than the third quantity, determine a fourth quantity of target used objects with the earliest storage time from used objects of the storage cluster, the fourth quantity being a difference between the third quantity and the quantity of objects corresponding to the second available logical unit.

[0159] Write the service data into the first quantity of available objects, the objects corresponding to the second available logical unit, and the fourth quantity of target used objects.

[0160] In a possible implementation, before determining the first quantity of available objects in the available object queue, the determining module 62 is further configured to:

[0161] In a case where it is detected that there is an abnormal logical unit in the first available logical unit, at least one third available logical unit other than the first available logical unit is determined from the storage cluster;

[0162] Objects corresponding to the at least one third available logical unit are added to the available object queue, and objects corresponding to the abnormal logical unit are deleted from the available object queue, a number of the objects corresponding to the at least one third available logical unit is greater than or equal to a number of the objects corresponding to the abnormal logical unit.

[0163] In a possible implementation, the processing module 61 is further configured to:

[0164] Objects corresponding to the abnormal logical unit are added to a to-be-repaired queue, and objects in the to-be-repaired queue are unavailable;

[0165] In a case where a number of objects included in the available object queue is less than a preset number, objects successfully repaired in the to-be-repaired queue are added to the available object queue.

[0166] In a possible implementation, the processing module 61 is further configured to:

[0167] In a case where a preset condition is met, a target object interval is determined in the storage cluster, and a preset number of objects are included in the target object interval;

[0168] According to the target object interval, the available object queue is updated;

[0169] The preset condition includes at least one of the following:

[0170] A number of objects included in the available object queue is less than or equal to a preset threshold;

[0171] All objects included in the available object queue are in an unavailable state;

[0172] The available object queue is emptied.

[0173] Figure 7 An example of a schematic diagram of an entity structure of an electronic device is shown in FIG. 1. ​As shown, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute a data writing method, which includes determining a first available logical unit from a storage cluster and adding an object corresponding to the first available logical unit to an available object queue; in response to a write instruction of service data, determining a first quantity of available objects in the available object queue; and performing a write operation of the service data according to the first quantity and a second quantity of available objects required by the service data.

[0174] In addition, the logical instruction in the memory 730 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or in other words the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0175] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the data writing method provided by each of the above embodiments, which includes: determining a first available logical unit from a storage cluster and adding an object corresponding to the first available logical unit to an available object queue; in response to a write instruction of service data, determining a first quantity of available objects in the available object queue; and performing a write operation of the service data according to the first quantity and a second quantity of available objects required by the service data.

[0176] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a data writing method provided by any of the above embodiments, and the method comprises: determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue; in response to a write instruction of service data, determining a first quantity of available objects in the available object queue; and performing a write operation of the service data according to the first quantity and a second quantity of available objects required by the service data.

[0177] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0178] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0179] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data writing method, characterized in that: include: Determine a first available logical unit from the storage cluster, and add an object corresponding to the first available logical unit to an available object queue; In response to a write instruction of the business data, determining a first number of available objects in the available object queue; executing a write operation of the business data according to the first quantity and a second quantity of available objects required by the business data; The performing the writing operation of the business data according to the first number and the second number of available objects required by the business data includes: When the first number is greater than or equal to the second number, the service data is written into an available object in the available object queue.

2. The method according to claim 1, characterized in that The determining a first available logical unit from the storage cluster and adding an object corresponding to the first available logical unit to an available object queue includes: Starting from the first logical unit in the storage cluster, determining the status of each logical unit in sequence, the status being available or unavailable; At least one object corresponding to the first available logical unit in an available state is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

3. The method according to claim 1 or 2, characterized in that The performing the writing operation of the business data according to the first number and the second number of available objects required by the business data further includes: When the first number is less than the second number, determine a second available logical unit other than the first available logical unit from the storage cluster; determine a third number of available objects corresponding to the second available logical unit, and add the third number of available objects to the available object queue to obtain an updated available object queue, where the third number is equal to the difference between the first number and the second number; and write the business data into the available objects in the updated available object queue.

4. The method according to claim 3, characterized in that The method further comprises: If the first number is smaller than the second number and the number of objects corresponding to the second available logical unit is smaller than the third number, determining a fourth number of target used objects with the earliest storage time from among the used objects in the storage cluster, where the fourth number is the difference between the third number and the number of objects corresponding to the second available logical unit; The business data is written into the first number of available objects, objects corresponding to the second available logical units, and the fourth number of target used objects.

5. The method according to claim 1 or 2, characterized in that Before determining the first number of available objects in the available object queue, the method further includes: In a case where an abnormal logical unit is detected in the first available logical units, determining at least one third available logical unit other than the first available logical unit from the storage cluster; Adding objects corresponding to the at least one third available logical unit to the available object queue, and deleting objects corresponding to the abnormal logical unit in the available object queue, wherein the number of objects corresponding to the at least one third available logical unit is greater than or equal to the number of objects corresponding to the abnormal logical unit.

6. The method according to claim 5, characterized in that The method further comprises: Adding the object corresponding to the abnormal logical unit to a queue to be repaired, where the object in the queue to be repaired is unavailable; When the number of objects included in the available object queue is less than a preset number, the successfully repaired objects in the to-be-repaired queue are added to the available object queue.

7. The method according to claim 1 or 2, characterized in that The method further comprises: If a preset condition is met, determining a target object interval in the storage cluster, where the target object interval includes a preset number of objects; updating the available object queue according to the target object interval; The preset conditions include at least one of the following: The number of objects included in the available object queue is less than or equal to a preset threshold; All objects included in the available object queue are in an unavailable state; The objects in the available object queue are cleared.

8. A device for implementing the data writing method according to any one of claims 1 to 7, characterized in that: include: a processing module, configured to determine a first available logical unit from a storage cluster, and add an object corresponding to the first available logical unit to an available object queue; a determining module, configured to determine a first number of available objects in the available object queue in response to a write instruction of the business data; A writing module is used to perform a writing operation of the business data according to the first quantity and a second quantity of available objects required by the business data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data writing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data writing method according to any one of claims 1 to 7 is implemented.

Images