Data writing method, device and equipment

Abstract

The invention provides a data writing method, device and equipment, and the method comprises the steps: determining a first available logic unit from a storage cluster, and adding an object corresponding to the first available logic unit into an available object queue; in response to a writing instruction of business data, determining a first number of available objects in the available object queue; and according to the first quantity and the second quantity of the available objects required by the business data, executing a write-in operation of the business data. By writing the service data into the object of the storage cluster, the situation that data storage is abnormal and data cannot be accessed due to the fact that the number of redundant resources is limited is avoided, continuous writing of the service data is achieved, and the reliability of data storage is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data writing method, device and equipment. Background Art

[0002] Errors are prone to occur during the data storage process, which has an adverse impact on the storage stability of the data.

[0003] In order to provide data storage stability, a redundant protection strategy is currently generally adopted, that is, data is stored by setting certain redundant resources. These redundant resources may include, for example, redundant copies, redundant storage media, redundant nodes, and the like.

[0004] However, the number of redundant resources is limited. When the number of offline resources exceeds the number of redundant resources during data storage, data storage exceptions may still occur. Summary of the invention

[0005] The present application provides a data writing method, device and equipment to improve the reliability of data storage.

[0006] In a first aspect, the present application provides a data writing method, comprising:

[0007] Determine a first available logical unit from the storage cluster, and add an object corresponding to the first available logical unit to an available object queue;

[0008] In response to a write instruction of the business data, determining a first number of available objects in the available object queue;

[0009] The write operation of the business data is performed according to the first number and a second number of available objects required by the business data.

[0010] In a possible implementation, determining a first available logical unit from a storage cluster and adding an object corresponding to the first available logical unit to an available object queue includes:

[0011] Starting from the first logical unit in the storage cluster, determining the status of each logical unit in turn, the status being available or unavailable;

[0012] At least one object corresponding to the first available logical unit whose status is available is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

[0013] In a possible implementation, performing the writing operation of the business data according to the first number and a second number of available objects required by the business data includes:

[0014] When the first number is greater than or equal to the second number, writing the service data into an available object in the available object queue;

[0015] When the first number is less than the second number, determine a second available logical unit other than the first available logical unit from the storage cluster; determine a third number of available objects corresponding to the second available logical unit, and add the third number of available objects to the available object queue to obtain an updated available object queue, wherein the third number is equal to the difference between the first number and the second number; and write the business data into the available objects in the updated available object queue.

[0016] In a possible implementation, the method further includes:

[0017] When the first number is less than the second number, and the number of objects corresponding to the second available logical unit is less than the third number, determining a fourth number of target used objects with the earliest storage time among the used objects in the storage cluster, where the fourth number is a difference between the third number and the number of objects corresponding to the second available logical unit;

[0018] The business data is written into the first number of available objects, objects corresponding to the second available logical units, and the fourth number of target used objects.

[0019] In a possible implementation, before determining the first number of available objects in the available object queue, the method further includes:

[0020] In the case where it is detected that there is an abnormal logical unit in the first available logical unit, determining at least one third available logical unit other than the first available logical unit from the storage cluster;

[0021] The objects corresponding to the at least one third available logical unit are added to the available object queue, and the objects corresponding to the abnormal logical unit in the available object queue are deleted, and the number of objects corresponding to the at least one third available logical unit is greater than or equal to the number of objects corresponding to the abnormal logical unit.

[0022] In a possible implementation, the method further includes:

[0023] Adding the object corresponding to the abnormal logic unit to a queue to be repaired, where the object in the queue to be repaired is unavailable;

[0024] When the number of objects included in the available object queue is less than a preset number, the successfully repaired objects in the to-be-repaired queue are added to the available object queue.

[0025] In a possible implementation, the method further includes:

[0026] In the case where a preset condition is met, determining a target object interval in the storage cluster, wherein the target object interval includes a preset number of objects;

[0027] updating the available object queue according to the target object interval;

[0028] The preset condition includes at least one of the following:

[0029] The number of objects included in the available object queue is less than or equal to a preset threshold;

[0030] All objects included in the available object queue are in an unavailable state;

[0031] The objects in the available object queue are cleared.

[0032] In a second aspect, the present application provides a data writing device, comprising:

[0033] A processing module, configured to determine a first available logical unit from a storage cluster, and add an object corresponding to the first available logical unit to an available object queue;

[0034] A determination module, configured to determine a first number of available objects in the available object queue in response to a write instruction of the business data;

[0035] A writing module is used to perform a writing operation of the business data according to the first number and a second number of available objects required by the business data.

[0036] In a possible implementation manner, the processing module is specifically configured to:

[0037] Starting from the first logical unit in the storage cluster, determining the status of each logical unit in turn, the status being available or unavailable;

[0038] At least one object corresponding to the first available logical unit whose status is available is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

[0039] In a possible implementation, the writing module is specifically used for:

[0040] When the first number is greater than or equal to the second number, writing the service data into an available object in the available object queue;

[0041] When the first number is less than the second number, determine a second available logical unit other than the first available logical unit from the storage cluster; determine a third number of available objects corresponding to the second available logical unit, and add the third number of available objects to the available object queue to obtain an updated available object queue, wherein the third number is equal to the difference between the first number and the second number; and write the business data into the available objects in the updated available object queue.

[0042] In a possible implementation, the writing module is further used for:

[0043] When the first number is less than the second number, and the number of objects corresponding to the second available logical unit is less than the third number, determining a fourth number of target used objects with the earliest storage time among the used objects in the storage cluster, where the fourth number is a difference between the third number and the number of objects corresponding to the second available logical unit;

[0044] The business data is written into the first number of available objects, objects corresponding to the second available logical units, and the fourth number of target used objects.

[0045] In a possible implementation manner, before determining the first number of available objects in the available object queue, the determining module is further configured to:

[0046] In the case where it is detected that there is an abnormal logical unit in the first available logical unit, determining at least one third available logical unit other than the first available logical unit from the storage cluster;

[0047] The objects corresponding to the at least one third available logical unit are added to the available object queue, and the objects corresponding to the abnormal logical unit in the available object queue are deleted, and the number of objects corresponding to the at least one third available logical unit is greater than or equal to the number of objects corresponding to the abnormal logical unit.

[0048] In a possible implementation manner, the processing module is further configured to:

[0049] Adding the object corresponding to the abnormal logic unit to a queue to be repaired, where the object in the queue to be repaired is unavailable;

[0050] When the number of objects included in the available object queue is less than a preset number, the successfully repaired objects in the to-be-repaired queue are added to the available object queue.

[0051] In a possible implementation manner, the processing module is further configured to:

[0052] In the case where a preset condition is met, determining a target object interval in the storage cluster, wherein the target object interval includes a preset number of objects;

[0053] updating the available object queue according to the target object interval;

[0054] The preset condition includes at least one of the following:

[0055] The number of objects included in the available object queue is less than or equal to a preset threshold;

[0056] All objects included in the available object queue are in an unavailable state;

[0057] The objects in the available object queue are cleared.

[0058] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data writing method as described in any one of the first aspects is implemented.

[0059] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data writing method as described in any one of the first aspects.

[0060] The data writing method, device and equipment provided by the present application first determine the first available logical unit from the storage cluster, and add the object corresponding to the first available logical unit to the available object queue. After receiving the write instruction of the business data, in response to the write instruction of the business data, determine the first number of available objects in the available object queue, and perform the write operation of the business data according to the first number and the second number of available objects required by the business data. By writing the business data to the objects of the storage cluster, the situation in which data storage anomalies and data inaccessibility occur due to the limited number of redundant resources is avoided, the continuous writing of business data is achieved, and the reliability of data storage is improved. Since the available objects are added to the available object queue in advance, the frequency of real-time requests for objects to the storage cluster during the process of writing business data to the objects is also reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0063] Figure 2 A schematic diagram of a data writing method according to an embodiment of the present application;

[0064] Figure 3 A schematic diagram of the process of writing business data provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of full coverage processing of data provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of an exception handling of a logic unit provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of the structure of a data writing device provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] First, the relevant concepts involved in this application are briefly introduced.

[0071] OSD: object-based storage device, which corresponds to the disk on the server. OSD is a storage medium at the hardware level, used to store all data and objects in the cluster. A disk or partition can serve as an OSD.

[0072] PG: placement group, refers to a collection of objects. A PG corresponds to one or more objects. At the logical level, objects are stored in PGs, and there is a certain mapping relationship between PGs and OSDs.

[0073] With the development of science and technology, various types of data are continuously generated. Some of these data can be discarded after use, but other data need to be stored for a long time for subsequent reading and use.

[0074] Taking video data as an example, video data records a wealth of information. By storing video data, when you need to obtain relevant information in the video data later, you can read it from the storage medium in a timely manner, which is very convenient. Therefore, it is of great significance to ensure the stability of storage devices in data storage services and have a safe redundant protection mechanism.

[0075] Currently in the field of cloud storage, redundant protection strategies such as multiple copies, correction and erasure redundancy, and cluster disaster recovery are mainly used to improve storage stability. The essence of redundant protection strategy is to protect data storage by setting certain redundant resources.

[0076] From the perspective of data, multiple copies are a solution to store multiple copies of the same data on a resource or storage device. Since multiple copies of the same data are stored, even if some of the data is abnormal, other normal data copies can be retrieved normally from the resource or storage device.

[0077] Erasure redundancy is a solution that improves storage stability from the perspective of storage media by creating an erasure strategy. When storing data, in addition to setting up resources or storage devices for storing data, redundant resources or redundant storage devices are also set up. When an abnormality occurs in the resource or storage device storing data, the data on the abnormal resource or storage device is written to the redundant resource or redundant storage device, thereby ensuring the continuity of data services and the normal playback of historical data services.

[0078] Cluster disaster recovery is a solution that provides storage stability from the perspective of nodes in the cluster by setting up redundant nodes. In a cluster, if some nodes go down, the redundant nodes can take over the work of the downed nodes, including storing the data in the redundant nodes, thereby ensuring the normal operation of the business.

[0079] However, whether setting up redundant copies, redundant storage media or redundant nodes, the number is limited. When the number of offline resources exceeds the number of redundant resources, data storage will still be abnormal.

[0080] For example, for the multi-copy strategy, if the number of abnormal data exceeds the number of redundant data, it is impossible to obtain data by reading the redundant data; for example, for the erasure redundancy strategy, if the number of failed resources or devices exceeds the number of redundant resources or redundant storage devices, it is impossible to guarantee that the data stored in all failed resources or devices can be written to the corresponding redundant resources or redundant storage devices; for example, for the cluster disaster recovery strategy, if the number of downed nodes exceeds the number of redundant nodes, it is impossible to guarantee that the services of all downed nodes can be taken over by the corresponding redundant nodes.

[0081] That is, through the analysis of the above resource protection strategies, when the redundancy protection strategy exceeds the set upper limit, the cluster status will be abnormal, and both historical object data and new business data will be at risk of being inaccessible and inwriteable.

[0082] Based on this, the embodiment of the present application provides a data writing method, which combines the real-time file system (RTFS) and the storage cluster management system to ensure that the business continues to read and write without interruption when the index storage pool is downgraded or under normal circumstances after the data storage pool redundancy strategy fails, in view of the fact that the current data protection strategy may cause data loss and the inability to write new business data. The scheme of the embodiment of the present application will be introduced in conjunction with the accompanying drawings.

[0083] First combine Figure 1 An applicable application scenario of the embodiment of the present application is introduced.

[0084] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, it includes an RTFS file system and a storage cluster, wherein the RTFS file system belongs to the business layer application, and the storage cluster belongs to the underlying storage medium. The RTFS file system and the storage cluster can be deployed on the same device or on different devices, and the embodiments of the present application do not limit this.

[0085] A storage cluster includes multiple logical units, such as Figure 1 The example shows three logical units, namely PG1, PG2 and PG3. Each logical unit maps one or more OSDs. Figure 1 In the example, multiple OSDs are shown, including OSD1, OSD2, ..., OSDn. Each OSD corresponds to one or more objects, such as Figure 1 In the example, obj1, obj2, ..., obj32, etc. Therefore, each logical unit corresponds to one or more objects.

[0086] The RTFS file system and the storage cluster can interact with each other through the Reliable Autonomic Distributed Object Store (RADOS) interface. For example, the RTFS file system can send a write request to the storage cluster through the RADOS interface, request an object from the storage cluster, write data to the object in the storage cluster, and the storage cluster returns the request result to the RTFS file system through the RADOS interface.

[0087] In the above embodiments, Figure 1 An applicable application scenario of the present application is introduced. Figure 1 Example application scenarios, combined with Figure 2 The solution of the embodiment of the present application is introduced.

[0088] Figure 2 A schematic diagram of a data writing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0089] S21, determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue.

[0090] The storage cluster includes at least one logical unit. For any logical unit, the logical unit may be an available logical unit or an unavailable logical unit. The available logical unit refers to a logical unit in a normal state, and the unavailable logical unit refers to a logical unit in an abnormal state. Each logical unit corresponds to one or more objects. If the logical unit is an available logical unit, the object currently corresponding to the logical unit is an available logical object.

[0091] Objects in the storage cluster can be used for data writing. In order to improve the reliability of data writing, an available object queue can be constructed. Specifically, first, a first available logical unit is determined in the logical units included in the storage cluster. The number of the first available logical units is one or more, and the first available logical unit is a subset of the available logical units included in the storage cluster, which can be all the current available logical units in the storage cluster, or a part of the current available logical units in the storage cluster.

[0092] In the storage cluster, since the first available logical unit is a normal logical unit, the object corresponding to the first available logical unit is also a currently available object. After determining the first available logical unit, the object corresponding to the first available logical unit is added to the available object queue, and the available object queue includes several currently available objects.

[0093] Optionally, in the embodiment of the present application, the size of each object is the same, and the type of data to be written is also known, so the maximum number of objects included in the available object queue is also certain. Taking video data as an example, the size of video data generally does not exceed 2G, then the available object queue can be set to include a maximum of 32 objects, and the size of each object is 64MB.

[0094] For any first available logical unit, all objects corresponding to the first available logical unit may be added to the available object queue, or some objects corresponding to the first available logical unit may be added to the available object queue, which is not limited in this embodiment of the present application.

[0095] S22: In response to a write instruction of the business data, determine a first number of available objects in an available object queue.

[0096] After receiving a write instruction for the business data, it is necessary to determine a first number of available objects in the available object queue and a second number of available objects required by the business data.

[0097] For any object in the available objects, the object is an available object when it is initially added to the available object queue, but since the object may not be used immediately, the state of the object may change. For example, the object may be abnormal due to some reasons, resulting in unavailability. For example, other parallel businesses may have used the object, resulting in a decrease in the number of available objects in the available object queue. Therefore, before writing business data, it is necessary to determine the first number of available objects in the available object queue, and the first number is greater than or equal to 0.

[0098] S23: Execute a write operation of the business data according to the first quantity and a second quantity of available objects required by the business data.

[0099] The second number of available objects required for business data refers to the number of objects required to write business data into objects in the storage cluster. Since the size of each object is fixed and the size of business data is also known, the second number of available objects required for business data can be determined based on the size of the business data and the size of the object, and the second number is greater than or equal to 1.

[0100] After determining the first quantity and the second quantity, the writing operation of the business data can be performed according to the relationship between the first quantity and the second quantity. The first quantity may be greater than or equal to the second quantity, or may be less than the second quantity. If the first quantity is greater than or equal to the second quantity, it indicates that the available objects in the available object queue are sufficient for writing the business data, so the business data is directly written to the available objects in the available object queue; if the first quantity is less than the second quantity, it indicates that the available objects in the available object queue are insufficient for writing the business data, so it is necessary to additionally request the storage cluster to allocate the corresponding objects for writing the business data.

[0101] The data writing method provided in the embodiment of the present application first determines the first available logical unit from the storage cluster, and adds the object corresponding to the first available logical unit to the available object queue. After receiving the write instruction of the business data, in response to the write instruction of the business data, the first number of available objects in the available object queue is determined, and the write operation of the business data is performed according to the first number and the second number of available objects required by the business data. By writing the business data to the objects of the storage cluster, the situation in which data storage anomalies and data inaccessibility occur due to the limited number of redundant resources is avoided, the continuous writing of business data is achieved, and the reliability of data storage is improved. Since the available objects are added to the available object queue in advance, the frequency of real-time requests for objects to the storage cluster during the process of writing business data to the objects is also reduced.

[0102] Based on any of the above embodiments, the solution provided in the embodiments of the present application is further introduced below in conjunction with the accompanying drawings.

[0103] Figure 3 A schematic diagram of the business data writing process provided in the embodiment of the present application is as follows: Figure 3 As shown, the storage cluster includes multiple logical units, such as PG1, PG2 and PG3. Each PG maps a corresponding OSD, for example, PG1 maps OSD1 and OSD3, PG2 maps OSD2 and OSD4, and PG3 maps PDS5 and OSDn (partial correspondence is not exhaustive).

[0104] First, you need to determine the first available logical unit from the storage cluster and add the object corresponding to the first available logical unit to the available object queue. Figure 3 This section introduces the process.

[0105] In one possible implementation, starting from the first logical unit in the storage cluster, the status of each logical unit is determined in turn, which is either available or unavailable, and then at least one object corresponding to the first available logical unit whose status is available is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

[0106] by Figure 3 For example, assuming that PG1, PG2 and PG3 are the first logical unit, the second logical unit and the third logical unit respectively, first determine whether the status of PG1 is available. If the status of PG1 is available, PG1 is determined as a first available logical unit, and then the object corresponding to PG1 is added to the available object queue.

[0107] After adding the object corresponding to PG1 to the available object queue, determine whether the number of objects in the available object queue is equal to the preset number. If not, continue to determine whether the state of the second logical unit is available, that is, whether the state of PG2 is available. If the state of PG2 is unavailable, continue to determine whether the state of the third logical unit, that is, PG3, is available.

[0108] If the status of PG3 is available, PG3 is also determined as a first available logical unit, and then the object corresponding to PG3 is added to the available object queue. If the number of objects corresponding to PG3 is greater than the difference between the number of objects currently included in the available object queue and the preset number, some of the objects corresponding to PG3 can be selected to be added to the available object queue. For example, the preset number is 32, the available object queue currently includes 30 objects, and the number of objects corresponding to PG3 is 5, then 2 of the 5 objects corresponding to PG3 can be selected and added to the available object queue. When the number of objects included in the available object queue reaches the preset number, adding new objects to the available object queue can be suspended.

[0109] The number of objects included in the available object queue changes dynamically. After an available object is added to the available object queue, the number of available objects in the available object queue will gradually decrease because other services may need to use the available object.

[0110] Therefore, after receiving a business data write instruction and before writing the business data, it is necessary to determine the first number of available objects in the available object queue, and perform the business data write operation based on the first number and the second number of available objects required by the business data.

[0111] When the first number is greater than or equal to the second number, the business data is written into the available objects in the available object queue, that is, a second number of available objects are selected from the first number of available objects, and the business data is written into the second number of available objects.

[0112] In the case where the first number is less than the second number, since the available objects in the available object pair column are not sufficient to realize the writing of business data, it is necessary to determine the second available logical unit other than the first available logical unit from the storage cluster. Then, the third number of available objects corresponding to the second available logical unit is determined, and the third number of available objects is added to the available object queue to obtain an updated available object queue, and the third number is equal to the difference between the first number and the second number. After the third number of available objects is added to the available object queue, the number of available objects included in the updated available object queue is the sum of the first number and the third number, and the third number is equal to the difference between the first number and the second number, so the sum of the first number and the third number is equal to the second number, that is, the number of available objects included in the updated available object queue is greater than or equal to the second number. In this case, the second number of available objects is determined in the updated available object queue, and the business data is written to the available objects in the updated available object queue.

[0113] In the process of writing business data, in the case where the first number is less than the second number, since the available objects included in the available object queue are insufficient to write the business data, it is necessary to additionally request the storage cluster to allocate a third number of available objects to add to the available object queue. However, in some cases, the number of objects corresponding to the second available logical unit is less than the third number, so that even if all the remaining available objects in the storage cluster are added to the available object queue, it is still insufficient to write the business data. In view of this situation where the number of available objects is insufficient, in order to avoid the loss of business data writing, the business data can be written by full coverage processing of the data. Figure 4 This section introduces the process.

[0114] Figure 4 A schematic diagram of full coverage processing of data provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the storage cluster includes multiple logical units, such as PG1, PG2 and PG3, and each PG includes corresponding objects.

[0115] When a first number of available objects included in the available object queue is less than a second number of available objects required for business data, and the number of objects corresponding to the second available logical unit is less than a third number, a fourth number of target used objects with the earliest storage time can be determined from the used objects in the storage cluster, wherein the fourth number is the difference between the third number and the number of objects corresponding to the second available logical unit.

[0116] by Figure 4For example, the first available logical unit includes PG1 and PG3, and the second available logical unit is PG2. The first number of available objects in the current available object queue is 22, and the second number of available objects required by the business data is 28. It is necessary to request 6 available objects from the storage cluster, that is, the third number is equal to 6. However, the number of objects corresponding to PG2 is 3, that is, Figure 4 In this case, it is necessary to determine the fourth number of target used objects in the used objects of the storage cluster. In the embodiment of the present application, the used objects refer to the objects that have been written with data. The fourth number is the difference between the third number and the number of objects corresponding to the second available logical unit. Figure 4 In the example, the third number is 6, and the number of objects corresponding to PG2 is 3, so the fourth number is equal to 3. The number of used objects included in the storage cluster may exceed the fourth number. In this case, the used object with the earliest storage time is preferentially selected as the target used object. The storage time of the object refers to the time when the data is written to the object. The used object with the earliest storage time is the object to which the data is written earliest.

[0117] After determining the fourth number of target used objects, the business data is written into the first number of available objects, the objects corresponding to the second available logical unit, and the fourth number of target used objects. Figure 4 For example, the target used objects include obj1, obj2 and obj3, then the first number of available objects is the 22 available objects included in the current available object queue, and the objects corresponding to the second available logical unit are Figure 4 The objects corresponding to PG2 in the example are obj6, obj7 and obj8, and the fourth number of target used objects include obj1, obj2 and obj3, which together constitute the objects used to write business data.

[0118] For the target used object, other data was originally written into the target used object. In order to ensure that the newly written business data is not interrupted, full overwrite deletion is initiated, and the used object with the earliest storage time is given priority to release storage space to write business data. However, in the case of storage cluster abnormalities, the RTFS file system cannot perceive the status of abnormal objects in the storage cluster. Therefore, when the number of available objects recorded in the RTFS file system is sufficient, but the number of available objects in the storage cluster is insufficient, data write loss will occur. Therefore, in the process of writing business data, only when the number of available objects in the available object queue is insufficient and it is impossible to request enough available objects from the storage cluster, will the full overwrite operation be performed to write business data through the used objects.

[0119] After the object corresponding to the first available logical unit is added to the available object queue, the first available logical unit may be abnormal, and the object corresponding to the abnormal first available logical unit will also be abnormal. The abnormal object is unavailable, so relevant abnormal processing is required.

[0120] Specifically, before determining the first number of available objects in the available object queue, it is detected whether there is an abnormal logical unit in the first available logical unit.

[0121] In the case where an abnormal logical unit is detected in the first available logical unit, at least one third available logical unit other than the first available logical unit may be determined from the storage cluster, and then objects corresponding to the at least one third available logical unit are added to an available object queue, and objects corresponding to the abnormal logical unit in the available object queue are deleted. The number of objects corresponding to the at least one third available logical unit is greater than or equal to the number of objects corresponding to the abnormal logical unit.

[0122] Combine the following Figure 5 This section describes the exception handling process.

[0123] Figure 5 The following is a schematic diagram of an exception handling of a logic unit provided in an embodiment of the present application, as shown in FIG. Figure 5 As shown, the storage cluster includes multiple logical units, such as PG1, PG2 and PG3, and each PG includes corresponding objects.

[0124] Before determining the first number of available objects in the available object queue, the abnormal logical unit may be determined in combination with the CRUSH algorithm, so that when writing service data, the service data is not written into the object corresponding to the abnormal logical unit.

[0125] For example, Figure 5 In the example, PG1 is a first available logical unit, and the objects corresponding to PG1 include obj9 and obj10. When PG1 is detected as an abnormal logical unit, at least one third available logical unit needs to be determined from the storage cluster. Figure 5 PG2 in , then add objects obj6, obj7 and obj8 corresponding to PG2 to the available object queue, and remove obj9 and obj10 from the available object queue.

[0126] Furthermore, for the abnormal logical unit, the object corresponding to the abnormal logical unit can be added to the queue to be repaired for repair. The objects in the queue to be repaired are unavailable objects. Figure 5 For example, objects obj9 and obj10 corresponding to PG1 are added to the queue to be repaired for repair.

[0127] When the number of objects included in the available object queue is less than the preset number, the successfully repaired objects in the queue to be repaired can be added to the available object queue. For example, after adding obj9 and obj10 to the queue to be repaired, if the repair is successful, when the number of available object queues is less than the preset number, obj9 and obj10 are added to the available object queue in time for writing of business data. By adding the successfully repaired objects to the available object queue in time, the frequency of requesting available objects from the storage cluster can be reduced, thereby improving the efficiency of data storage.

[0128] For the storage cluster, all OSDs are storage data resources, but the storage cluster cannot determine the validity of object data. Therefore, the business layer can issue corresponding structure rules, and the storage cluster returns the corresponding target object interval according to the corresponding structure rules, so that the business layer can update the available object queue.

[0129] Specifically, when a preset condition is met, a target object interval is determined in the storage cluster, and the available object queue is updated according to the target object interval, wherein the target object interval includes a preset number of objects, and the preset condition includes at least one of the following:

[0130] Condition 1: The number of objects included in the available object queue is less than or equal to a preset threshold.

[0131] When the available object queue is initially constructed, it includes a preset number of available objects. However, as business data is written, the number of available objects included in the available object queue gradually decreases. When the number of available objects included in the available object queue is less than or equal to a preset threshold, for example, the available objects in the available object queue are used up, or the number of available objects is less than 5, the available object queue can be updated.

[0132] Condition 2: All objects included in the available object queue are in an unavailable state.

[0133] When an object is initially added to the available object queue, the object is available, but due to some reasons, the object is abnormal and the state of the object becomes unavailable. When all objects included in the available object queue are in the unavailable state, the available object queue needs to be updated.

[0134] Condition 3: The objects in the available object queue are cleared.

[0135] When the business server system is restarted, all objects in the available object queue will be cleared, and the available object queue does not include any objects. In this case, the available object queue needs to be updated.

[0136] For example, if the target object range is testObj_1 to testObj_32, the number of objects included in the target object range is 32. Then the business layer sends a structure rule to the storage cluster. The structure rule example is as follows:

[0137] {

[0138] 'baseKey':'testObj';

[0139] 'obj_range':0,32;

[0140] 'map_used':4032(00111111000000)

[0141] 'policy':'free'

[0142] …

[0143] }

[0144] After receiving the structure rule, the storage cluster obtains the objects in the target object range of testObj_1 to testObj_32 and returns them to the business layer so that the business layer can update the available object queue.

[0145] In summary, the data writing method provided in the embodiment of the present application reduces the frequency of the business layer requesting available objects from the storage cluster by requesting available objects in the storage cluster in advance and adding them to the available object queue. There is no need to request available objects in real time during the process of writing business data, thereby reducing the performance load of the device and improving the data storage capacity. Since the objects added to the available object queue are available objects, the reliability of data storage is further guaranteed, and the continuous writing of business data is realized.

[0146] The data writing device provided by the present application is described below. The data writing device described below and the data writing method described above can refer to each other.

[0147] Figure 6 A schematic diagram of the structure of a data writing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device comprises:

[0148] The processing module 61 is used to determine a first available logical unit from the storage cluster, and add an object corresponding to the first available logical unit to an available object queue;

[0149] A determination module 62, configured to determine a first number of available objects in the available object queue in response to a write instruction of the business data;

[0150] The writing module 63 is used to perform a writing operation of the business data according to the first number and a second number of available objects required by the business data.

[0151] In a possible implementation, the processing module 61 is specifically configured to:

[0152] Starting from the first logical unit in the storage cluster, determining the status of each logical unit in turn, the status being available or unavailable;

[0153] At least one object corresponding to the first available logical unit whose status is available is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

[0154] In a possible implementation, the writing module 63 is specifically used for:

[0155] When the first number is greater than or equal to the second number, writing the service data into an available object in the available object queue;

[0156] When the first number is less than the second number, determine a second available logical unit other than the first available logical unit from the storage cluster; determine a third number of available objects corresponding to the second available logical unit, and add the third number of available objects to the available object queue to obtain an updated available object queue, wherein the third number is equal to the difference between the first number and the second number; and write the business data into the available objects in the updated available object queue.

[0157] In a possible implementation, the writing module 63 is further used for:

[0158] When the first number is less than the second number, and the number of objects corresponding to the second available logical unit is less than the third number, determining a fourth number of target used objects with the earliest storage time among the used objects in the storage cluster, where the fourth number is a difference between the third number and the number of objects corresponding to the second available logical unit;

[0159] The business data is written into the first number of available objects, objects corresponding to the second available logical units, and the fourth number of target used objects.

[0160] In a possible implementation, before determining the first number of available objects in the available object queue, the determining module 62 is further configured to:

[0161] In the case where an abnormal logical unit is detected in the first available logical unit, determining at least one third available logical unit other than the first available logical unit from the storage cluster;

[0162] The objects corresponding to the at least one third available logical unit are added to the available object queue, and the objects corresponding to the abnormal logical unit in the available object queue are deleted, and the number of objects corresponding to the at least one third available logical unit is greater than or equal to the number of objects corresponding to the abnormal logical unit.

[0163] In a possible implementation, the processing module 61 is further configured to:

[0164] Adding the object corresponding to the abnormal logic unit to a queue to be repaired, where the object in the queue to be repaired is unavailable;

[0165] When the number of objects included in the available object queue is less than a preset number, the successfully repaired objects in the to-be-repaired queue are added to the available object queue.

[0166] In a possible implementation, the processing module 61 is further configured to:

[0167] In the case where a preset condition is met, determining a target object interval in the storage cluster, wherein the target object interval includes a preset number of objects;

[0168] updating the available object queue according to the target object interval;

[0169] The preset condition includes at least one of the following:

[0170] The number of objects included in the available object queue is less than or equal to a preset threshold;

[0171] All objects included in the available object queue are in an unavailable state;

[0172] The objects in the available object queue are cleared.

[0173] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the data writing method, which includes: determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue; determining a first number of available objects in the available object queue in response to a write instruction for business data; and executing a write operation of the business data according to the first number and a second number of available objects required by the business data.

[0174] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0175] On the other hand, the present application also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data writing method provided by the above-mentioned embodiments, and the method includes: determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue; in response to a write instruction of business data, determining a first number of available objects in the available object queue; and performing a write operation of the business data based on the first number and a second number of available objects required for the business data.

[0176] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the data writing method provided in the above-mentioned embodiments, the method comprising: determining a first available logical unit from a storage cluster, and adding an object corresponding to the first available logical unit to an available object queue; in response to a write instruction of business data, determining a first number of available objects in the available object queue; and performing a write operation of the business data based on the first number and a second number of available objects required for the business data.

[0177] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data writing method, characterized in that: include: Determine a first available logical unit from the storage cluster, and add an object corresponding to the first available logical unit to an available object queue; In response to a write instruction of the business data, determining a first number of available objects in the available object queue; The write operation of the business data is performed according to the first number and a second number of available objects required by the business data.

2. The method according to claim 1, characterized in that The determining a first available logical unit from the storage cluster and adding an object corresponding to the first available logical unit to an available object queue includes: Starting from the first logical unit in the storage cluster, determining the status of each logical unit in turn, the status being available or unavailable; At least one object corresponding to the first available logical unit whose status is available is added to the available object queue until the number of objects in the available object queue is equal to a preset number.

3. The method according to claim 1 or 2, characterized in that: The performing the writing operation of the business data according to the first number and the second number of available objects required by the business data includes: When the first number is greater than or equal to the second number, writing the service data into an available object in the available object queue; When the first number is less than the second number, determine a second available logical unit other than the first available logical unit from the storage cluster; determine a third number of available objects corresponding to the second available logical unit, and add the third number of available objects to the available object queue to obtain an updated available object queue, wherein the third number is equal to the difference between the first number and the second number; and write the business data into the available objects in the updated available object queue.

4. The method according to claim 3, characterized in that The method further comprises: When the first number is less than the second number, and the number of objects corresponding to the second available logical unit is less than the third number, determining a fourth number of target used objects with the earliest storage time among the used objects in the storage cluster, where the fourth number is a difference between the third number and the number of objects corresponding to the second available logical unit; The business data is written into the first number of available objects, objects corresponding to the second available logical units, and the fourth number of target used objects.

5. The method according to claim 1 or 2, characterized in that: Before determining the first number of available objects in the available object queue, the method further includes: In the case where an abnormal logical unit is detected in the first available logical unit, determining at least one third available logical unit other than the first available logical unit from the storage cluster; The objects corresponding to the at least one third available logical unit are added to the available object queue, and the objects corresponding to the abnormal logical unit in the available object queue are deleted, and the number of objects corresponding to the at least one third available logical unit is greater than or equal to the number of objects corresponding to the abnormal logical unit.

6. The method according to claim 5, characterized in that The method further comprises: Adding the object corresponding to the abnormal logic unit to a queue to be repaired, where the object in the queue to be repaired is unavailable; When the number of objects included in the available object queue is less than a preset number, the successfully repaired objects in the to-be-repaired queue are added to the available object queue.

7. The method according to claim 1 or 2, characterized in that: The method further comprises: In the case where a preset condition is met, determining a target object interval in the storage cluster, wherein the target object interval includes a preset number of objects; updating the available object queue according to the target object interval; The preset condition includes at least one of the following: The number of objects included in the available object queue is less than or equal to a preset threshold; All objects included in the available object queue are in an unavailable state; The objects in the available object queue are cleared.

8. A data writing device, characterized in that: include: A processing module, configured to determine a first available logical unit from a storage cluster, and add an object corresponding to the first available logical unit to an available object queue; A determination module, configured to determine a first number of available objects in the available object queue in response to a write instruction of the business data; A writing module is used to perform a writing operation of the business data according to the first number and a second number of available objects required by the business data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data writing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data writing method according to any one of claims 1 to 7 is implemented.

Images