Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a storage medium, and the method comprises the steps: responding to a recovery operation of a user on at least one first data volume which is not used, and merging the first data volume into a second data volume; a target state of each first data object in the second data volume is obtained, a target data recovery bit table is generated according to the target state, a target state bit in the target data recovery bit table records the target state of the first data object, the target state bit is marked as 0 if the target state is an unrecovered state, and the target state bit is marked as 1 if the target state is a recovered state; according to a preset timed recovery task, deleting the first data object marked as 0, and marking the corresponding target status bit as 1; when it is determined that the read-write operation exists on the second data volume, different read-write strategies for each first data object are determined according to different target status bits in the target data recovery bit table, and the utilization rate of storage resources is increased.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] With the rapid development of technologies such as cloud computing and big data, the amount of data has exploded, placing higher demands on the capacity, performance, and flexibility of storage systems. Distributed block storage systems have become an important choice for enterprise-level data storage due to their scalability, high availability, and data consistency. However, in actual applications, with business adjustments, data migration or deletion, a large number of data volumes are marked as "deleted" or "no longer in use". If these resources cannot be recovered and reused in a timely and effective manner, storage space will be seriously wasted and maintenance costs will increase.

[0003] In the prior art, the processing methods for deleting data volumes are mostly limited to directly deleting the data volume to release space immediately, or deleting the data volume and then putting it into the recycle bin and then deleting it with a delay to release space slowly. It should be noted that when deleting a data volume, the contents of the entire data volume need to be deleted at one time to complete the space release of the data volume. It can be seen that it takes a certain amount of time to delete a data volume, and only after the deletion is completed can the data volume be set and data written to the released space.

[0004] Therefore, when there is a situation where the used data volume capacity and the amount of written data are large, the "deleted" or "no longer used" data volume needs to be completely released before the data is written. However, this process takes a lot of time and fails to fully utilize the potential value of the "deleted" or "no longer used" data volume. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a data processing method, apparatus, device and storage medium to solve the problem that it takes a lot of time to release "deleted" or "no longer used" data volumes and fails to fully utilize their potential value. The specific technical solution is as follows: In a first aspect of the present invention, a data processing method is provided, the method comprising: In response to a user's recovery operation on at least one first data volume that has stopped being used, merging at least one first data volume into a new second data volume; Acquire a plurality of first data objects of the second data volume and a target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records a target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; According to the preset timed recycling task, first data objects whose target status bits are marked as 0 are deleted in sequence, and target status bits corresponding to the deleted first data objects are marked as 1; When it is determined that there is a read / write operation on the second data volume, a different read / write strategy for each first data object is determined according to different target status bits in the target data recovery bit table.

[0006] Optionally, in response to a user's recycling operation on at least one first data volume that has stopped being used, merging at least one first data volume into a new second data volume includes: In response to a user's recovery operation on at least one first data volume that has stopped being used, obtaining a second data object and a first capacity of each first data volume; determining a second capacity of a second data volume to be established according to a sum of at least one of the first capacities; Allocate free space corresponding to the second capacity from the distributed block storage system; The second data volume is created in the free space, and the at least one second data object of the first data volume is migrated to the second data volume to obtain the first data object of the second data volume.

[0007] Optionally, acquiring a plurality of first data objects of the second data volume and a target state of each first data object, and generating a target data recycling bit table in the second data volume according to the target state, includes: Acquire a plurality of first data objects of the second data volume; Obtaining a target capacity size of each of the first data objects of the second data volume and a second capacity of the second data volume; Determining a target number of first data objects in the second data volume by using the target capacity and the second capacity, wherein the first data objects include a third data object carrying data and a fourth data object with empty data, and the target capacity of each of the first data objects is the same; generating, for the second data volume, an initial data recovery bit table including a first state bit of the target bit number according to the target number; Establishing a one-to-one association relationship between the first data object and the first status bit; Acquire a target state of each first data object in the second data volume; According to the target state and the association relationship, a value of the first state bit of the initial data recovery bit table is marked to obtain a target data recovery bit table including a target state bit that records the target state of the first data object.

[0008] Optionally, in the step of sequentially deleting the first data objects whose target status bits are marked as 0 according to the preset timed recycling task, and marking the target status bits corresponding to the deleted first data objects as 1, including: When it is determined that the data service in the distributed block storage system is idle, a scheduled recovery task is started; According to a preset timed recycling task, traverse the target data recycling bit table in the second data volume; In the case where it is determined that the target state bit in the target data recovery bit table is marked as 1, skipping the first data object whose target state bit is marked as 1, and continuing to determine the mark value of the next target state bit; When it is determined that the target status bit in the target data recovery bit table is marked as 0, the first data object whose target status bit is marked as 0 is deleted, and the target status bit corresponding to the deleted first data object is marked as 1.

[0009] Optionally, in the case where it is determined that the target status bit in the target data recovery bit table is marked as 0, after deleting the first data object whose target status bit is marked as 0 and marking the target status bit corresponding to the deleted first data object as 1, the method further includes: In the case where it is determined that the scheduled recovery task is interrupted, recording the traversal position in the target data recovery bit table when the task is interrupted; When it is determined that the timed recycling task is restarted, the traversal is continued backward from the traversal position.

[0010] Optionally, when it is determined that there is a read / write operation on the second data volume, determining a different read / write strategy for each first data object according to different target status bits in the target data recovery bit table includes: In a case where it is determined that a read operation is performed on the second data volume, determining a second state bit marked as 0 and a third state bit marked as 1 in the target data recovery bit table, wherein the second state bit and the third state bit are both target state bits; dividing the first data object in the second data volume into a fifth data object corresponding to the second status bit and a sixth data object corresponding to the third status bit; If the fifth data object is read, 0 is returned; If the sixth data object is read, the target data in the sixth data object is returned.

[0011] Optionally, when it is determined that there is a read / write operation on the second data volume, determining a different read / write strategy for each first data object according to different target status bits in the target data recovery bit table further includes: In the case where it is determined that a write operation exists on the second data volume, determining a fourth state bit marked as 0 and a fifth state bit marked as 1 in the target data recovery bit table, wherein both the fourth state bit and the fifth state bit belong to target state bits; dividing the first data object in the second data volume into a seventh data object corresponding to the fourth status bit and an eighth data object corresponding to the fifth status bit; If a write operation is performed in the first space corresponding to the seventh data object, the data to be written is combined with the value of the first space and then filled into the first space, and the fourth status bit is modified to 1; If a write operation is performed in the second space corresponding to the eighth data object, the data to be written is directly filled into the second space.

[0012] In a second aspect of the present invention, there is also provided a data processing device, comprising: a merging module, configured to merge at least one first data volume into a new second data volume in response to a user's recovery operation on at least one first data volume that has stopped being used; a first generating module, configured to obtain a plurality of first data objects of the second data volume and a target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records a target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; a deleting module, configured to delete the first data objects whose target status bits are marked as 0 in sequence according to a preset timed recycling task, and mark the target status bits corresponding to the first data objects after deletion as 1; The first determination module is used to determine different read and write strategies for each first data object according to different target status bits in the target data recovery bit table when it is determined that there is a read and write operation on the second data volume.

[0013] In a third aspect of the implementation of the present invention, there is also provided a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; The processor is used to read the program in the memory to implement and execute any of the above-mentioned data processing methods.

[0014] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the above-mentioned data processing methods.

[0015] The data processing method provided by the embodiment of the present invention responds to the user's recovery operation on at least one first data volume that has been discontinued, merges at least one first data volume into a new second data volume, realizes the recovery and reuse of the discontinued data volume, effectively improves the utilization rate of storage resources, reduces storage fragmentation, obtains a plurality of first data objects of the second data volume and the target state of each first data object, and generates a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records the target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; according to the preset timed recovery task, the first data objects whose target state bits are marked as 0 are sequentially recovered. The target data recovery bit table is used to recover the data object and the target data recovery bit table is used to recover the data object. The ... BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0017] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention; Figure 2 for Figure 1 The flowchart of step 101 in a data processing provided by an embodiment of the present invention is shown; Figure 3 for Figure 1 The flowchart of step 103 in a data processing provided by an embodiment of the present invention is shown; Figure 4 for Figure 1 A flowchart of steps of a read-write operation offset algorithm in data processing provided by an embodiment of the present invention is shown; Figure 5 is a structural schematic diagram of a fault handling device provided by an embodiment of the present invention; Figure 6 It is a structural diagram of a communication device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0019] Reference Figure 1 , shows a flow chart of steps of a data processing method provided by an embodiment of the present invention, the method may include: Step 101: In response to a user's recovery operation on at least one first data volume that is no longer in use, merge at least one first data volume into a new second data volume.

[0020] In the embodiment of the present invention, the user can select at least one first data volume that has been stopped from use in the management interface or the recycle bin for recycling operation. To determine that the selected data volume is stopped from use, it is necessary to add a stop-use mark to the data volume, and then the user selects the first data volume through this mark. To achieve this purpose, the distributed block storage system collects and analyzes the usage of several target data volumes, which may include the read and write frequency of the data volume, the data access mode, the load condition, etc., and determines the time point when each target data volume was last accessed or used based on the collected usage. By comparing the current time with the last use time of each target data volume, the latest unused time of each data volume is calculated, that is, the length of time since the last use, and a first time threshold is set. This threshold represents the length of time that the data volume is considered to be "stopped in use". If the latest unused time of a target data volume is greater than this first time threshold, it is considered that the data volume is no longer frequently used. For those target data volumes whose latest unused time is greater than the first time threshold, a stop-use target mark is added to them. This mark can be used for subsequent management decisions, such as the recovery, migration or adjustment of the backup strategy of the data volume. The specific steps include: Obtain usage status of several target data volumes in a distributed block storage system; Determining the most recent usage time of the target data volumes according to the usage situation; Determine the latest unused time of the target data volumes according to the most recent used time; When it is determined that the latest unused time length of any target data volume is greater than the first time threshold, a stop-use target identifier is added to any target data volume.

[0021] By identifying and marking data volumes that are no longer active, it is easier to make management decisions based on this identification.

[0022] In the embodiment of the present invention, after the user selects at least one first data volume, a data volume recycling operation is performed. The recycling operation first merges at least one first data volume. During the merging process, the system merges the capacity of one or more first data volumes to be recycled and allocates them to a new data volume (corresponding to the second data volume of the present application). The capacity of the second data volume is the sum of the capacities of all the first data volumes after the merger. The source data volume will no longer be visible in the system, and its space will be occupied by the second data volume.

[0023] Further, step 101, such as Figure 2 As shown: Step 1011 : in response to a user's recovery operation on at least one first data volume that is no longer in use, obtain a second data object and a first capacity of each first data volume.

[0024] Step 1012: Determine a second capacity of a second data volume to be created according to the sum of the at least one first capacity.

[0025] Step 1013: allocate free space corresponding to the second capacity from the distributed block storage system.

[0026] Step 1014: create a second data volume in the free space, and migrate at least one second data object of the first data volume to the second data volume to obtain a first data object of the second data volume.

[0027] For example, the user selects two first data volumes for recycling, namely, data volume A and data volume B. The capacity of data volume A is 100 GB, including second data objects A1 and A2 with data, and an idle second data object A3. The capacity of data volume B is 50 GB, including second data objects B1 and B2 with data, and an idle second data object B3. The data objects (A1, A2, A3, B1, B2, B3) and capacities (100 GB and 50 GB) of data volumes A and B are obtained, and the capacity of the new second data volume is calculated to be 100 GB + 50 GB = 150 GB. A new data volume is created in the free space, called data volume C, with a capacity of 150 GB. The data objects (A1, A2, A3, B1, B2, B3) in data volumes A and B are migrated to data volume C.

[0028] It should be noted that after the first data volume is merged, it will no longer be visible in the system. At this time, the identifiers (such as name, path, etc.) on the first data volume are removed or marked as unavailable in the system, so users can no longer access the first data volume through these identifiers. However, the data of the first data volume is not actually deleted, but migrated to the second data volume. Through the above operation, the system administrator or storage management system only needs to manage the new second data volume, and does not need to manage each first data volume separately, which reduces storage fragmentation and simplifies the management of storage resources.

[0029] Step 102, obtain several first data objects of the second data volume and the target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records the target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1.

[0030] In order to ensure the consistency of subsequent data read and write operations in the embodiment of the present invention, a target data recovery bit table is generated for the merged second data volume. The size of the target data recovery bit table is calculated according to the capacity of the second data volume and the size of the first data object therein. Each bit of the target data recovery bit table represents the state of a first data object. The specific steps include: Obtaining a target capacity size of each first data object in the second data volume and a second capacity of the second data volume; Determine the target number of first data objects in the second data volume according to the target capacity and the second capacity, wherein the first data objects include a third data object carrying data and a fourth data object with empty data, and the target capacity of each first data object is the same; generating an initial data recovery bit table including a first state bit of the target number of bits for the second data volume according to the target number; Establishing a one-to-one association relationship between the first data object and the first status bit; Acquire a plurality of first data objects of the second data volume and a target state of each first data object; According to the target state and the association relationship, the value of the first state bit of the initial data recovery bit table is marked to obtain a target data recovery bit table including a target state bit that records the target state of the first data object.

[0031] For example, the target capacity of each first data object in the second data volume is 2G, and the second capacity of the second data volume is 20G, so the target number of first data objects is 20G÷2G=10. Therefore, the set initial data recovery bit table has 10 target bits of first state bits, namely bit0~9, and a one-to-one association relationship is established between the first data objects and the first state bits, that is, bit0 corresponds to first data object 0, bit1 corresponds to first data object 1, bit2 corresponds to first data object 2, bit3 corresponds to first data object 3...bit9 corresponds to first data object 9, and then the target state of each first data object is obtained, and the first state bit is numerically marked according to the target state. The target state refers to the state of whether the first data object is recovered. When the merger is just performed, all the first data objects are not recovered, so the values ​​set on bits0~9 of the target data recovery bit table are all 0.

[0032] Through the above steps, the data objects in the second data volume can be effectively managed and optimized, and the status of the first data object can be tracked and controlled through the status bit to facilitate data recovery, migration or deletion operations. The target data recovery bit table can be used as a basis for system decision-making, helping system administrators or automation tools to make more reasonable data management decisions.

[0033] Step 103 : According to the preset timed recycling task, first data objects whose target status bits are marked as 0 are deleted in sequence, and target status bits corresponding to the deleted first data objects are marked as 1.

[0034] In the embodiment of the present invention, after the first data volume is merged, it is necessary to recycle the content of the first data volume. The prior art will perform overall deletion and release during recycling, while the present application performs partial recycling based on the target status bit mark in the target data recycling bit table. During recycling, according to the data service situation of the data center, a timed recycling task is set when the service is idle to accelerate data recycling. The target data recycling bit table of the data volume is traversed through the timed recycling task process. If the data object status bit is marked as 1, it indicates that the data object has been recycled, and the data object is skipped. If the data object status bit is marked as 0, it indicates that the data object has not been recycled. At this time, the data object will be deleted and the status bit will be set to 1.

[0035] Further, step 103, such as Figure 3 As shown: Step 1031 , when it is determined that the data service in the distributed block storage system is idle, start a scheduled recycling task.

[0036] Step 1032: traverse the target data recovery bit table in the second data volume according to the preset timed recovery task.

[0037] Step 1033, when it is determined that the target state bit in the target data recovery bit table is marked as 1, skip the first data object whose target state bit is marked as 1, and continue to determine the mark value of the next target state bit.

[0038] Step 1034 , when it is determined that the target status bit in the target data recovery bit table is marked as 0, the first data object whose target status bit is marked as 0 is deleted, and the target status bit corresponding to the deleted first data object is marked as 1.

[0039] Through the above process, data objects that are no longer needed are regularly cleaned up to release storage space and improve the efficiency of the storage system. By using the target data recovery bit table, the system can selectively recycle data objects, and each recovery takes a short time, so the released space can be put into use as soon as possible, improving the efficiency of storage space use.

[0040] In the embodiment of the present invention, the scheduled recovery task is performed when the data service is idle. However, during the execution of the scheduled recovery task, if a new data service appears, the scheduled recovery task will be interrupted and the new data service will be executed first. The interrupted scheduled recovery task will record the traversal position in the target data recovery bit table when it was interrupted, and then when the scheduled recovery task is resumed, it will continue to traverse backward from the traversal position. The specific steps also include: When it is determined that the scheduled recovery task is interrupted, the traversal position in the target data recovery bit table when it is interrupted is recorded; When it is determined that the scheduled recycling task is restarted, continue to traverse backwards from the traversal position.

[0041] Among them, the traversal is carried out according to the occupancy of the target status bits in the target data recovery bit table. For example, the target data recovery bit table includes bit0~9, 10 target status bits, and the target status bits are traversed in the order of bit0, bit1, bit2, bit3, bit4, bit5, bit6, bit7, bit8, bit9 from small to large or bit9, bit8, bit7, bit6, bit5, bit4, bit3, bit2, bit1, bit0 from large to small.

[0042] The above steps can ensure the continuity and integrity of the data recovery task, and even if the task is interrupted, it can be restored from the interruption point to avoid duplication of work and omissions. In this way, the application can manage storage resources more efficiently while ensuring the accuracy of data recovery operations.

[0043] Step 104 : when it is determined that there is a read / write operation on the second data volume, determine a different read / write strategy for each first data object according to different target status bits in the target data recovery bit table.

[0044] In the embodiment of the present invention, after part of the first data object is recycled through a timed recycling task, the recycled space will be reused. When performing read and write operations on the second data volume, different operation strategies will be performed according to the identification of the target status bit. For example, when performing a read operation on the merged second data volume, the target data recycling bit table in the second data volume will be read first, and then the specific mark of each target status bit therein will be determined. If the mark is 0, it indicates that the first data object has not been recycled. At this time, the data is stopped from use, does not exist or is invalid. At this time, all 0 data will be returned. If the mark is 1, it indicates that the first data object has been recycled. At this time, the stored first data object is a new data object after the update. At this time, the target data can be directly read when reading. The specific steps include: When it is determined that there is a read operation on the second data volume, a second state bit marked as 0 and a third state bit marked as 1 in the target data recovery bit table are determined, and both the second state bit and the third state bit belong to the target state bit; dividing the first data object in the second data volume into a fifth data object corresponding to the second status bit and a sixth data object corresponding to the third status bit; If the fifth data object is read, 0 is returned; If the sixth data object is read, the target data in the sixth data object is returned.

[0045] It should be noted that after deletion and recycling, the target status bit will also be marked as 1, but at this time the recycled space is empty and has not been written. There is no data when reading at this time, but it will not return 0 at this time, but will return the preset target field, such as 00001233, indicating that this space has been recycled but not used yet. If the target data stored in the recycled space is 0, in order to distinguish it from the output of the unrecycled first data object, when reading the unrecycled first data object, 0 will be returned and an unrecycled mark will be added. In the above manner, it is ensured that the read operation is only performed on valid data objects, while avoiding reading data that has been marked as deleted or invalid. This helps to improve the accuracy of data reading and the overall performance of the system.

[0046] In addition, when performing a write operation, this embodiment will also perform different operation strategies according to the identification of the target status bit. That is, for a data object whose target status bit is 0, the system first regards its content as all-0 data, and merges the new data with the all-0 data before writing it to avoid offset of the written data. After writing is completed, the corresponding data object status bit is modified to 1, indicating that this data object has been recycled and the data at this position has been updated. For a data object whose target status bit is 1, the original data is directly overwritten. The specific steps include: When it is determined that there is a write operation on the second data volume, a fourth state bit marked as 0 and a fifth state bit marked as 1 in the target data recovery bit table are determined, and both the fourth state bit and the fifth state bit belong to the target state bit; dividing the first data object in the second data volume into a seventh data object corresponding to the fourth status bit and an eighth data object corresponding to the fifth status bit; If a write operation is performed in the first space corresponding to the seventh data object, the data to be written is combined with the value of the first space and then filled into the first space, and the fourth status bit is modified to 1; If a write operation is performed in the second space corresponding to the eighth data object, the data to be written is directly filled into the second space.

[0047] Furthermore, when performing read and write operations on the second data volume, the metadata information of the second data volume will be added to the memory first. Then, when performing read and write operations, the target read and write offset and target read and write length will be determined first, and then the internal read and write offset of the corresponding mapped first data volume will be determined based on this information. At the same time, the read and write offset offs and length len of the second data volume will be compared with the capacity of the first data volume. When the target read and write offset and target read and write length are within the capacity of a first data volume, it is converted to the reading and writing of a first data volume. When the target read and write offset and target read and write length exceed the capacity of one or more first data volumes, it is converted to the reading and writing of two or more first data volumes. Taking the conversion to a maximum of 2 first data volumes as an example, the specific algorithm is as follows. Figure 4 As shown, there are several first data volumes, whose capacities are a0, a1, a2, ..., an, respectively. To determine the target read / write offset offs and the target read / write length len for the second data volume, firstly, the initial capacity sum is set to a0, sum=a0 (a0 is the capacity of the first data volume 1). If the target read / write offset is greater than a0 (offs>sum), it is determined that the read / write operation will not be performed on a0. At this time, the capacity sum=the capacity of the first data volume 1+the capacity of the first data volume 2. Then, this capacity sum is compared with the target read / write offset until the target read / write offset is less than the capacity sum. At this time, the first offset of the first data volume 1=the target read / write offset-(capacity sum-the capacity of the first data volume 2), that is, offs1=offs-(sum-ai), and offs1 is the first offset of the first data volume 1. If the target read / write offset is less than the initial capacity and a0 (offs < sum), it is determined that the read / write operation can be performed on a0. At this time, the offset of the first data volume 1 on a0 is equal to the target read / write offset. At this time, it is further determined whether the sum of the offset of the first data volume 1 and the target read / write length exceeds a0, that is, offs1 + len > sum. If not, the read / write operation will be completed in the first data volume 1. The first read / write length len1 = len. len1 is the first read / write length of the first data volume 1. If it exceeds a0, it means that the read / write operation When operating on a0 and a1, part of the data will be read on a0 first. The first read / write length is the capacity of the first data volume 1 - the first offset, that is, len1=sum-offs1. Then the remaining part will be read on the first data volume 2, a1. Reading and writing on a1 starts from the beginning, so the second offset of the first data volume 2 at this time is offs2=0. The second read / write length is the target read / write length - (capacity of the first data volume 1 - first offset), len2=-offs+len-sum, and len2 is the second read / write length.

[0048] For example, there are two first data volumes, namely, data volume A and data volume B, with capacities of 100GB and 50GB respectively. The second data volume C is formed by merging data volumes A and B, with a total capacity of 150GB. If the user requests to perform a read / write operation on the new data volume C, the offset is 30GB and the length is 20GB. Since 30GB + 20GB = 50GB, this range is completely within the capacity of data volume A (100GB), so this read / write operation can be directly mapped to the internal read / write offset of data volume A, that is, starting from 30GB, read and write 20GB of data. If the user requests to perform a read / write operation on the new data volume C, the offset is 80GB and the length is 40GB. Since 80GB + 40GB = 120GB, this range exceeds the capacity of data volume A (100GB), so segmented processing is required. The first 20GB (from 80GB to 100GB) is mapped to the internal read / write offset of data volume A, that is, starting from 80GB, read and write 20GB of data. The last 20GB (from 100GB to 120GB) is mapped to the internal read / write offset of data volume B, that is, starting from 0GB, 20GB of data is read and written. If the user requests to perform read / write operations on the new data volume C, the offset is 110GB and the length is 20GB. Since 110GB>100GB, no read / write operations will be performed on data volume A, and the internal read / write offset mapped to data volume B is 10GB. At this time, starting from 10GB, 20GB of data (from 10GB to 30GB) is read and written.

[0049] In this way, the read and write operations of the new data volume can be correctly mapped to the source data volumes that make it up, ensuring the continuity and correctness of the data.

[0050] The data processing method provided by the embodiment of the present invention responds to the user's recovery operation on at least one first data volume that has been discontinued, merges at least one first data volume into a new second data volume, realizes the recovery and reuse of the discontinued data volume, effectively improves the utilization rate of storage resources, reduces storage fragmentation, obtains a plurality of first data objects of the second data volume and the target state of each first data object, and generates a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records the target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; according to the preset timed recovery task, the first data objects whose target state bits are marked as 0 are sequentially recovered. The target data recovery bit table is used to recover the data object and the target data recovery bit table is used to recover the data object. The ...

[0051] Reference Figure 5 , which shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention, such as Figure 5 As shown, the device may include: The merging module 201 is configured to merge at least one first data volume into a new second data volume in response to a user's recovery operation on at least one first data volume that is no longer in use.

[0052] The first generation module 202 is used to obtain a plurality of first data objects of the second data volume and a target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records a target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1.

[0053] The deleting module 203 is used to delete the first data objects whose target status bits are marked as 0 in sequence according to a preset timed recycling task, and mark the target status bits corresponding to the deleted first data objects as 1.

[0054] The first determination module 204 is configured to determine different read / write strategies for each first data object according to different target status bits in the target data recovery bit table when it is determined that there is a read / write operation on the second data volume.

[0055] Optionally, the merging module 201 specifically includes: The first acquisition submodule is used to acquire the second data object and the first capacity of each first data volume in response to a user's recovery operation on at least one first data volume that has stopped being used.

[0056] The first determining submodule is configured to determine a second capacity of a second data volume to be established according to the sum of at least one first capacity.

[0057] The first partitioning submodule is used to partition free space corresponding to the second capacity from the distributed block storage system.

[0058] The migration submodule is used to create a second data volume in the free space and migrate at least one second data object of the first data volume to the second data volume to obtain the first data object of the second data volume.

[0059] Optionally, the first generating module 202 specifically includes: The second acquisition submodule is used to acquire the target capacity size of each first data object in the second data volume and the second capacity of the second data volume.

[0060] The second determination submodule is used to determine the target number of first data objects in the second data volume according to the target capacity size and the second capacity, the first data objects including the third data object carrying data and the fourth data object with empty data, and the target capacity size of each first data object is the same.

[0061] The first generating submodule is used to generate an initial data recovery bit table including a first state bit of the target number of bits for the second data volume according to the target number.

[0062] The relationship establishing submodule is used to establish a one-to-one association relationship between the first data object and the first status bit.

[0063] The third acquisition submodule is used to acquire a plurality of first data objects of the second data volume and a target state of each first data object.

[0064] The marking submodule is used to mark the value of the first state bit of the initial data recovery bit table according to the target state and the association relationship, so as to obtain a target data recovery bit table including a target state bit recording the target state of the first data object.

[0065] Optionally, the deletion module 203 specifically includes: The third determining submodule is used to start the scheduled recovery task when it is determined that the data service in the distributed block storage system is idle.

[0066] The first traversal submodule is used to traverse the target data recovery bit table in the second data volume according to a preset timed recovery task.

[0067] The fourth determination submodule is used to skip the first data object whose target state bit is marked as 1 and continue to determine the mark value of the next target state bit when determining that the target state bit in the target data recovery bit table is marked as 1.

[0068] The deletion submodule is used to delete the first data object whose target status bit is marked as 0 when determining that the target status bit in the target data recovery bit table is marked as 0, and mark the target status bit corresponding to the deleted first data object as 1.

[0069] The recording submodule is used to record the traversal position of the target data recovery bit table when it is interrupted when it is determined that the scheduled recovery task is interrupted.

[0070] The second traversal submodule is used to continue traversing backward from the traversal position when it is determined that the scheduled recycling task is restarted.

[0071] Optionally, the first determining module 204 specifically includes: The fifth determination submodule is used to determine the second state bit marked as 0 and the third state bit marked as 1 in the target data recovery bit table when it is determined that there is a read operation on the second data volume, and the second state bit and the third state bit are both target state bits.

[0072] The second division submodule is used to divide the first data object in the second data volume into a fifth data object corresponding to the second status bit and a sixth data object corresponding to the third status bit.

[0073] The first reading submodule is configured to return 0 if the fifth data object is read.

[0074] The second reading submodule is used to return the target data in the sixth data object if the sixth data object is read.

[0075] Optionally, the first determining module 204 specifically further includes: The sixth determination submodule is used to determine the fourth state bit marked as 0 and the fifth state bit marked as 1 in the target data recovery bit table when it is determined that there is a write operation on the second data volume, and the fourth state bit and the fifth state bit are both target state bits.

[0076] The third division submodule is used to divide the first data object in the second data volume into a seventh data object corresponding to the fourth status bit and an eighth data object corresponding to the fifth status bit.

[0077] The first write operation submodule is used to merge the data to be written with the value of the first space and fill the first space, and modify the fourth status bit to 1 if a write operation is performed in the first space corresponding to the seventh data object.

[0078] The second write operation submodule is used to directly fill the data to be written into the second space if a write operation is performed in the second space corresponding to the eighth data object.

[0079] The data processing method provided by the embodiment of the present invention responds to the user's recovery operation on at least one first data volume that has been discontinued, merges at least one first data volume into a new second data volume, realizes the recovery and reuse of the discontinued data volume, effectively improves the utilization rate of storage resources, reduces storage fragmentation, obtains a plurality of first data objects of the second data volume and the target state of each first data object, and generates a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records the target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; according to the preset timed recovery task, the first data objects whose target state bits are marked as 0 are sequentially recovered. The target data recovery bit table is used to recover the data object and the target data recovery bit table is used to recover the data object. The ...

[0080] The embodiment of the present invention also provides a communication device, such as Figure 6 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. Memory 303, used for storing computer programs; The processor 301 is used to execute the program stored in the memory 303 to implement the following steps: In response to a user's recovery operation on at least one first data volume that has stopped being used, merging at least one first data volume into a new second data volume; Acquire a plurality of first data objects of the second data volume and a target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records a target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; According to the preset timed recycling task, first data objects whose target status bits are marked as 0 are deleted in sequence, and target status bits corresponding to the deleted first data objects are marked as 1; When it is determined that there is a read / write operation on the second data volume, a different read / write strategy for each first data object is determined according to different target status bits in the target data recovery bit table.

[0081] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into a physical address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0082] The communication interface is used for communication between the above terminal and other devices.

[0083] The memory may include a random access memory (RAM), or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0084] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0085] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the data processing method of the aforementioned embodiment.

[0086] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0087] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. According to the above description, it is obvious that the structure required for constructing this type of device is. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is to disclose the best mode of the present invention.

[0088] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0089] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all of the features of the individual embodiments disclosed above. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0090] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0091] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0092] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0095] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0096] It should be noted that the various data-related processes in the embodiments of the present application are all carried out in compliance with the corresponding data protection laws and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

Claims

1. A data processing method, characterized in that: The method comprises: In response to a user's recovery operation on at least one first data volume that has stopped being used, merging at least one first data volume into a new second data volume; Acquire a plurality of first data objects of the second data volume and a target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records a target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; According to the preset timed recycling task, first data objects whose target status bits are marked as 0 are deleted in sequence, and target status bits corresponding to the deleted first data objects are marked as 1; When it is determined that there is a read / write operation on the second data volume, a different read / write strategy for each first data object is determined according to different target status bits in the target data recovery bit table.

2. The method according to claim 1, characterized in that The step of merging the at least one first data volume into a new second data volume in response to a user's reclaiming operation on the at least one first data volume that has stopped being used includes: In response to a user's recovery operation on at least one first data volume that has stopped being used, obtaining a second data object and a first capacity of each first data volume; determining a second capacity of a second data volume to be established according to a sum of at least one of the first capacities; Allocate free space corresponding to the second capacity from the distributed block storage system; The second data volume is created in the free space, and the at least one second data object of the first data volume is migrated to the second data volume to obtain the first data object of the second data volume.

3. The method according to claim 1, characterized in that The acquiring of a plurality of first data objects of the second data volume and a target state of each first data object, and generating a target data recovery bit table in the second data volume according to the target state, includes: Acquire a plurality of first data objects of the second data volume; Obtaining a target capacity size of each of the first data objects of the second data volume and a second capacity of the second data volume; Determining a target number of first data objects in the second data volume by using the target capacity and the second capacity, wherein the first data objects include a third data object carrying data and a fourth data object with empty data, and the target capacity of each of the first data objects is the same; generating, for the second data volume, an initial data recovery bit table including a first state bit of the target bit number according to the target number; Establishing a one-to-one association relationship between the first data object and the first status bit; Acquire a target state of each first data object in the second data volume; According to the target state and the association relationship, a value of the first state bit of the initial data recovery bit table is marked to obtain a target data recovery bit table including a target state bit that records the target state of the first data object.

4. The method according to claim 1, characterized in that: In the step of sequentially deleting the first data objects whose target status bits are marked as 0 according to the preset timed recycling task, and marking the target status bits corresponding to the first data objects after deletion as 1, including: When it is determined that the data service in the distributed block storage system is idle, a scheduled recovery task is started; According to a preset timed recycling task, traverse the target data recycling bit table in the second data volume; In the case where it is determined that the target state bit in the target data recovery bit table is marked as 1, skipping the first data object whose target state bit is marked as 1, and continuing to determine the mark value of the next target state bit; When it is determined that the target status bit in the target data recovery bit table is marked as 0, the first data object whose target status bit is marked as 0 is deleted, and the target status bit corresponding to the deleted first data object is marked as 1.

5. The method according to claim 4, characterized in that In the case where it is determined that the target state bit in the target data recovery bit table is marked as 0, after deleting the first data object whose target state bit is marked as 0 and marking the target state bit corresponding to the deleted first data object as 1, the method further includes: In the case where it is determined that the scheduled recovery task is interrupted, recording the traversal position in the target data recovery bit table when the task is interrupted; When it is determined that the timed recycling task is restarted, the traversal is continued backward from the traversal position.

6. The method according to claim 1, characterized in that The determining, when determining that there is a read / write operation on the second data volume, of a different read / write strategy for each first data object according to different target status bits in the target data recovery bit table includes: In a case where it is determined that a read operation is performed on the second data volume, determining a second state bit marked as 0 and a third state bit marked as 1 in the target data recovery bit table, wherein the second state bit and the third state bit are both target state bits; dividing the first data object in the second data volume into a fifth data object corresponding to the second status bit and a sixth data object corresponding to the third status bit; If the fifth data object is read, 0 is returned; If the sixth data object is read, the target data in the sixth data object is returned.

7. The method according to claim 1, characterized in that The step of determining, when determining that a read / write operation exists on the second data volume, a different read / write strategy for each first data object according to different target status bits in the target data recovery bit table further includes: In the case where it is determined that a write operation exists on the second data volume, determining a fourth state bit marked as 0 and a fifth state bit marked as 1 in the target data recovery bit table, wherein both the fourth state bit and the fifth state bit belong to target state bits; dividing the first data object in the second data volume into a seventh data object corresponding to the fourth status bit and an eighth data object corresponding to the fifth status bit; If a write operation is performed in the first space corresponding to the seventh data object, the data to be written is combined with the value of the first space and then filled into the first space, and the fourth status bit is modified to 1; If a write operation is performed in the second space corresponding to the eighth data object, the data to be written is directly filled into the second space.

8. A data processing device, characterized in that: include: a merging module, configured to merge at least one first data volume into a new second data volume in response to a user's recovery operation on at least one first data volume that has stopped being used; a first generating module, configured to obtain a plurality of first data objects of the second data volume and a target state of each first data object, and generate a target data recovery bit table in the second data volume according to the target state, wherein each target state bit in the target data recovery bit table records a target state of a first data object, and if the target state is an unrecovered state, the target state bit is marked as 0, and if the target state is a recovered state, the target state bit is marked as 1; a deleting module, configured to delete the first data objects whose target status bits are marked as 0 in sequence according to a preset timed recycling task, and mark the target status bits corresponding to the first data objects after deletion as 1; The first determination module is used to determine different read and write strategies for each first data object according to different target status bits in the target data recovery bit table when it is determined that there is a read and write operation on the second data volume.

9. A communication device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the steps in the data processing method as described in any one of claims 1-7.

10. A readable storage medium for storing a program, characterized in that: When the stored program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images