Method, device, equipment and storage medium for orderly listing data of storage system

Abstract

The present invention discloses a method, apparatus, device and computer-readable storage medium for orderly enumerating data of a storage system, and relates to the technical field of data storage. The method comprises: obtaining a current enumerated shard sequence of the storage system; reading target unenumerated data from a target shard according to the current enumerated shard sequence; if unenumerated data exists in the current enumerated shard sequence, updating the current enumerated shard sequence according to the unenumerated data in the current enumerated shard sequence; if unenumerated data does not exist in the current enumerated shard sequence, determining that the data enumeration of the storage system is completed; the present invention can arrange data in an orderly manner during the data reading process of the storage system, which is convenient for customers to use, and does not require sorting and arranging massive data after reading it out, thereby being able to realize orderly enumeration of data of the storage system with less memory overhead and computing overhead in the case of massive data, thereby improving user experience.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a method, device, equipment and computer-readable storage medium for orderly enumerating data in a storage system. Background Art

[0002] Currently, storage systems write data to shards when storing it. There are two methods for writing data to shards: First, a fixed number of shards is created in advance, and the stored data is hashed to a specific shard. When writing to the shards, the data on each shard is sorted. Second, data is not created in advance. Data is written to the first shard first. When the first shard is full, a second shard is created, and data is written to the second shard, and so on.

[0003] Because data in a storage system is hashed across all shards, existing methods for enumerating data in an ordered manner require reading a portion of the data from all shards, sorting it, processing that portion, then rereading a portion from all shards, and repeating this process. This consumes a significant amount of memory and computing resources when the number of shards in the storage system is large, leading to a sharp decline in storage system performance. Therefore, how to achieve ordered enumeration of storage system data with minimal memory and computing overhead, while improving user experience, in the face of massive amounts of data, is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, device, equipment and computer-readable storage medium for orderly enumeration of data in a storage system, which can realize orderly enumeration of data in a storage system with less memory overhead and computing overhead, thereby improving user experience.

[0005] To solve the above technical problems, the present invention provides a method for orderly enumerating data in a storage system, comprising:

[0006] Obtaining a current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shard, and the data in the shard is sorted by data size;

[0007] Read target unlisted data from a target shard according to the currently listed shard sequence; wherein the target shard is the first shard or the last shard in the currently listed shard sequence; the target unlisted data is unlisted data in the target shard that is smaller than unlisted data in other shards or unlisted data in the target shard that is larger than unlisted data in other shards, and the unlisted data is data that has not been enumerated and read;

[0008] If there is unlisted data in the current enumerated fragment sequence, then the current enumerated fragment sequence is updated according to the unlisted data in the current enumerated fragment sequence, and the step of reading the target unlisted data from the target fragment according to the current enumerated fragment sequence is performed;

[0009] If there is no unlisted data in the current enumerated fragment sequence, it is determined that the data enumeration of the storage system is completed.

[0010] In some embodiments, the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data in the shard, and the target unenumerated data is the unenumerated data in the target shard that is smaller than the unenumerated data in other shards.

[0011] In some embodiments, the current enumerated shard sequence is a shard sequence sorted from smallest to largest according to the smallest unenumerated data in the shards, and reading the target unenumerated data from the target shard according to the current enumerated shard sequence includes:

[0012] Determine the first fragment in the currently enumerated fragment sequence as the target fragment;

[0013] The target unlisted data is read from the target slice; wherein the target unlisted data is unlisted data in the target slice that is smaller than comparison data, and the comparison data is the smallest unlisted data in the second slice in the current listed slice sequence.

[0014] In some embodiments, the data in the shards are sorted from small to large, and the reading of the target unlisted data from the target shards includes:

[0015] Find the position of the positioning data in the target slice; wherein the positioning data is the first unlisted data in the data sorting of the target slice that is greater than the comparison data;

[0016] The unlisted data before the position of the positioning data in the target slice is determined as the target unlisted data, and the target unlisted data is read.

[0017] In some embodiments, after searching for the location of the positioning data in the target slice, the method further includes:

[0018] Determine whether the positioning data is smaller than the smallest unlisted data in the third fragment in the currently listed fragment sequence;

[0019] If so, swapping the storage locations of the positioning data and the comparison data, and executing the steps of determining the unlisted data before the location of the positioning data in the target slice as the target unlisted data, and reading the target unlisted data;

[0020] If not, the steps of determining the unlisted data before the position of the positioning data in the target slice as the target unlisted data and reading the target unlisted data are performed.

[0021] In some embodiments, searching for the location of the positioning data in the target slice includes:

[0022] The position of the positioning data in the target slice is found using a binary search method.

[0023] In some embodiments, obtaining a current enumerated shard sequence of the storage system includes:

[0024] After obtaining the data enumeration instruction of the storage system, determining the current shard sequence of the storage system as the current enumeration shard sequence;

[0025] Correspondingly, the method for orderly enumerating data further includes:

[0026] After the data in the current shard changes, the shards are sorted according to the size of the minimum data or the maximum data in each shard in the storage system, and the current shard sequence is updated; wherein the current shard is any of the shards.

[0027] The present invention also provides a device for orderly enumerating data in a storage system, comprising:

[0028] A sequence acquisition module is configured to acquire a current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shard, and the data in the shard is sorted by data size;

[0029] A data reading module is configured to read target unlisted data from a target shard according to a currently listed shard sequence; wherein the target shard is the first shard or the last shard in the currently listed shard sequence; the target unlisted data is unlisted data in the target shard that is smaller than unlisted data in other shards or unlisted data in the target shard that is larger than unlisted data in other shards, and the unlisted data is data that has not been enumerated and read;

[0030] a sequence updating module, configured to update the current enumerated fragment sequence according to the unenumerated data in the current enumerated fragment sequence if there is unenumerated data in the current enumerated fragment sequence, and send a start signal to the data reading module;

[0031] The completion determination module is used to determine that the data enumeration of the storage system is completed if there is no unenumerated data in the current enumerated fragment sequence.

[0032] The present invention also provides a device for orderly enumerating data in a storage system, comprising:

[0033] memory for storing computer programs;

[0034] A processor is configured to implement the steps of the method for orderly enumerating data in a storage system as described above when executing the computer program.

[0035] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for orderly enumerating data in the storage system as described above are implemented.

[0036] The present invention provides a method for orderly enumerating data of a storage system, comprising: obtaining a current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shards, and the data in the shards are sorted by data size; reading target unenumerated data from a target shard according to the current enumerated shard sequence; wherein the target shard is the first shard or the last shard in the current enumerated shard sequence; the target unenumerated data is unenumerated data in the target shard that is smaller than unenumerated data in other shards or unenumerated data in the target shard that is larger than unenumerated data in other shards, and the unenumerated data is data that has not been enumerated and read; if unenumerated data exists in the current enumerated shard sequence, updating the current enumerated shard sequence according to the unenumerated data in the current enumerated shard sequence, and executing the step of reading the target unenumerated data from the target shard according to the current enumerated shard sequence; if unenumerated data does not exist in the current enumerated shard sequence, determining that data enumeration of the storage system is complete;

[0037] As can be seen, the present invention reads the target unlisted data from the target shard according to the current shard enumeration sequence, thereby enabling orderly data arrangement during the data reading process of the storage system, facilitating user use and eliminating the need to sort and organize massive amounts of data after reading. This allows for orderly data enumeration in the storage system with minimal memory and computational overhead, thus improving the user experience. Furthermore, the present invention also provides a device, apparatus, and computer-readable storage medium for orderly data enumeration in a storage system, which also have the aforementioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 A flowchart of a method for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of data stored on a shard in a storage system;

[0041] Figure 3 A flowchart of another method for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a current shard sequence in which data is fully written on the first shard of another method for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the current shard sequence after data is written to the second shard in another method for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the current shard sequence after data is written to the third shard in another method for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of a current enumerated shard sequence after target unenumerated data is read in another method for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0046] Figure 8 A structural block diagram of a device for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0047] Figure 9 A schematic diagram of the structure of a device for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0048] Figure 10 A schematic diagram of the specific structure of a device for orderly enumerating data in a storage system provided by an embodiment of the present invention;

[0049] Figure 11 A schematic structural diagram of a computer-readable storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] Please refer to Figure 1 , Figure 1 This is a flowchart of a method for orderly enumerating data in a storage system provided by an embodiment of the present invention. The method may include:

[0052] Step 101: Obtain a current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shard, and the data in the shard is sorted by data size.

[0053] The present embodiment does not limit the specific type of the storage system. For example, the storage system in the present embodiment may be a distributed storage system, such as a distributed object storage system; the storage system may also be a centralized storage system.

[0054] It can be understood that the currently listed fragment sequence in this embodiment can be a sequence of fragments in the storage system that currently have data that has not been read and listed (i.e., unlisted data). The fragments in the currently listed fragment sequence can be sorted according to the size of the smallest unlisted data (i.e., smallest unlisted data) or the largest unlisted data (i.e., largest unlisted data) in the fragment; for example, the currently listed fragment sequence can be a fragment sequence sorted from small to large according to the smallest unlisted data in the fragment, that is, the smallest unlisted data of the previous fragment in the currently listed fragment sequence is smaller than the smallest unlisted data in the next fragment, and the smallest unlisted data of the first fragment is the smallest; the currently listed fragment sequence can also be a fragment sequence sorted from large to small according to the smallest unlisted data in the fragment, and this embodiment does not impose any restrictions on this.

[0055] Correspondingly, in this embodiment, the data of each file in each shard in the currently listed shard sequence are sorted by data size. For example, the data in the shard can be sorted from small to large, that is, the previous data in each shard is smaller than the next data, and the first data in each shard is the smallest; the data in the shard can be sorted from large to small, and this embodiment does not impose any restrictions on this.

[0056] It should be noted that the specific method for the processor to obtain the current enumerated fragment sequence of the storage system in this step can be set by the designer according to the practical scenario and user needs. For example, the processor can determine the current fragment sequence of the storage system as the current enumerated fragment sequence after obtaining the data enumeration instruction of the storage system; that is, after receiving the instruction for orderly enumeration of data in the storage system (i.e., the data enumeration instruction), the processor can directly determine the current fragment sequence in the storage system (i.e., the current fragment sequence) as the current enumerated fragment sequence; that is, the current fragment sequence can be a fragment sequence sorted by the size of the minimum data or the maximum data in the fragment, and the data in the fragment is sorted by the data size. The processor can also generate the current enumerated fragment sequence according to the fragments in the storage system after obtaining the data enumeration instruction of the storage system; wherein, the data in each fragment of the storage system is sorted by the data size; for example, the processor can sort the fragments in the storage system by the size of the minimum data or the maximum data in the fragment after obtaining the data enumeration instruction of the storage system to generate the current enumerated fragment sequence. This embodiment does not impose any restrictions on this.

[0057] Correspondingly, in this step, after the processor obtains the data enumeration instruction of the storage system, it can determine the current shard sequence of the storage system as the current enumerated shard sequence. In this case, the method provided by this embodiment can also include a process for generating and updating the current shard sequence. For example, after the data in the current shard changes, the processor can sort the shards according to the size of the minimum data or the maximum data in each shard in the storage system, and update the current shard sequence; wherein, the current shard is any shard in the storage system, and the data in each shard in the storage system is sorted by data size. For example, after the data in the current shard changes, the processor can sort the shards according to the size of the minimum data in each shard in the storage system, and update the current shard sequence; correspondingly, the current enumerated shard sequence can be a shard sequence sorted by the size of the minimum unenumerated data in the shard.

[0058] Step 102: Read target unlisted data from the target shard according to the current enumerated shard sequence; wherein the target shard is the first shard or the last shard in the current enumerated shard sequence; the target unlisted data is unlisted data in the target shard that is smaller than unlisted data in other shards or unlisted data in the target shard that is larger than unlisted data in other shards; unlisted data is data that has not been enumerated and read.

[0059] It can be understood that the target slice in this step can be the first slice or the last slice in the current enumerated slice sequence, that is, the slice where the smallest unlisted data or the largest unlisted data in the current enumerated slice sequence is located. The target unlisted data in this step can be the unlisted data in the target slice that needs to be enumerated and read this time; wherein, when the current enumerated slice sequence is a slice sequence sorted by the size of the smallest unlisted data in the slice, the target unlisted data can be the unlisted data in the target slice that is smaller than the unlisted data in other slices, that is, the target unlisted data of the target slice is smaller than the unlisted data in other slices (i.e., other slices) in the current enumerated slice sequence; when the current enumerated slice sequence is a slice sequence sorted by the size of the largest unlisted data in the slice, the target unlisted data can be the unlisted data in the target slice that is larger than the unlisted data in other slices, that is, the target unlisted data of the target slice is larger than the unlisted data in other slices (i.e., other slices) in the current enumerated slice sequence.

[0060] It should be noted that since the data in the storage system is hashed on all shards, the existing ordered data enumeration method reads a portion of the data from all shards when starting to enumerate the data, such as Figure 2 As shown in the figure, suppose two data are read from each shard first. The results read from the first shard (shard-1) are: a-file and d-file; the results read from the second shard (shard-2) are: b-file and f-file; the results read from the third shard (shard-3) are: e-file and j-file; ... and the results read from the last shard (shard-N) are: g-file and h-file. After sorting, a-file, b-file, d-file, and e-file are removed. Since the data read from shard-1 has been used up at this time, it is necessary to read some data from shard-1 again and repeat the operation until the data on all shards is read and sorted. This ordered data enumeration method can handle the situation where the number of shards is small, but when there are a large number of shards (such as hundreds of thousands of shards), it requires a large amount of memory resources and computing resources to sort and organize the data, resulting in a sharp decline in performance.

[0061] Correspondingly, in this embodiment, the processor can read the target unlisted data from the target shard according to the current enumerated shard sequence, and read the smaller or larger unlisted data (i.e., the target unlisted data) in the current storage system each time the data is read. It can complete the orderly arrangement of data during the data reading process of the storage system, and does not need to sort and organize the data after reading it out. Therefore, in the case of massive data, it can realize the orderly enumeration of data in the storage system with less memory overhead and computing overhead.

[0062] Among them, the specific method for the processor to read the target unlisted data from the target slice according to the current enumerated slice sequence in this step can be set by the designer according to the practical scenario and user needs. For example, when the current enumerated slice sequence is a slice sequence sorted from small to large according to the smallest unlisted data in the slice, the processor can determine the first slice in the current enumerated slice sequence as the target slice; read the target unlisted data from the target slice; wherein the target unlisted data is the unlisted data in the target slice that is smaller than the comparison data, and the comparison data is the smallest unlisted data in the second slice in the current enumerated slice sequence.

[0063] Correspondingly, when the current enumerated shard sequence is a shard sequence sorted from largest to smallest according to the smallest unenumerated data in the shards, the processor can determine the last shard in the current enumerated shard sequence as the target shard; read the target unenumerated data from the target shard; wherein the target unenumerated data is the unenumerated data in the target shard that is smaller than the comparison data, and the comparison data is the smallest unenumerated data in the second-to-last shard in the current enumerated shard sequence.

[0064] Correspondingly, when the current enumerated shard sequence is a shard sequence sorted from small to large according to the largest unlisted data in the shard, the processor can determine the first shard in the current enumerated shard sequence as the target shard; read the target unlisted data from the target shard; wherein the target unlisted data is the unlisted data in the target shard that is larger than the comparison data, and the comparison data is the largest unlisted data in the second shard in the current enumerated shard sequence.

[0065] It should be noted that in this step, the processor can read the target unlisted data in the target shard in sequence according to the size of the data. For example, if the current listed shard sequence is a shard sequence sorted by the size of the smallest unlisted data in the shard, the processor can read the target unlisted data in the target shard in sequence according to the order of data from small to large.

[0066] Furthermore, the method provided in this embodiment may also include a data exchange process between shards, so as to correct the storage position of the data stored in each shard during the process of orderly listing data in the storage system, so that the data on the shards can be continuously ordered to improve the subsequent listing speed. For example, when the current listing shard sequence is a shard sequence sorted by the size of the smallest unlisted data in the shard, the processor can exchange the storage position of the positioning data and the comparison data when the positioning data in the target shard is smaller than the smallest unlisted data in the comparison shard; wherein the positioning data is the smallest unlisted data in the target shard that is larger than the target unlisted data, the comparison shard can be the shard with the second smallest smallest unlisted data among the shards other than the target shard in the current listing shard sequence, and the comparison data can be the smallest unlisted data of the shard with the smallest smallest unlisted data among the shards other than the target shard in the current listing shard sequence. For example, when the current enumerated shard sequence is a shard sequence sorted from small to large according to the smallest unlisted data in the shard, the processor may swap the storage positions of the positioning data and the comparison data when the positioning data in the target shard is smaller than the smallest unlisted data in the third shard in the current enumerated shard sequence; wherein the target unlisted data is the unlisted data in the target shard that is smaller than the comparison data, the comparison data is the smallest unlisted data in the second shard in the current enumerated shard sequence, and the positioning data is the first unlisted data in the data sorting of the target shard that is larger than the comparison data.

[0067] Accordingly, when the current enumerated fragment sequence is a fragment sequence sorted by the size of the largest unlisted data in the fragment, the processor can swap the storage positions of the positioning data and the comparison data in the target fragment when the positioning data in the target fragment is larger than the largest unlisted data in the comparison fragment; wherein the positioning data is the smallest unlisted data in the target fragment that is smaller than the target unlisted data, the comparison fragment can be the fragment with the second largest largest unlisted data among the fragments other than the target fragment in the current enumerated fragment sequence, and the comparison data can be the largest unlisted data of the fragment with the largest largest unlisted data among the fragments other than the target fragment in the current enumerated fragment sequence.

[0068] Step 103: If there is unlisted data in the current enumerated fragment sequence, the current enumerated fragment sequence is updated according to the unlisted data in the current enumerated fragment sequence, and the process proceeds to step 102.

[0069] It can be understood that after the target unlisted data is read, the target unlisted data can be determined as listed data. In this step, the processor can, after the target unlisted data is read, determine that there are other unlisted data in the current listed fragment sequence. Then, the processor can sort the fragments again according to the size of the smallest unlisted data or the largest unlisted data in the fragments, update the current listed fragment sequence, and return to step 102 to continue reading the target unlisted data in the updated current listed fragment sequence.

[0070] Among them, the specific method for the processor to update the current enumerated fragment sequence according to the unlisted data in the current enumerated fragment sequence in this step can be set by the designer according to practical scenarios and user needs. For example, when the current enumerated fragment sequence is a fragment sequence sorted from small to large according to the smallest unlisted data in the fragment, the processor can determine whether the smallest unlisted data of the first fragment in the current enumerated fragment sequence is smaller than the smallest unlisted data of the second fragment after determining the target unlisted data as the enumerated data; if not, swap the sequence numbers of the first fragment and the second fragment in the current enumerated fragment sequence to update the current enumerated fragment sequence and enter step 102; if so, it can directly enter step 102.

[0071] Step 104: If there is no unlisted data in the current enumerated shard sequence, it is determined that the data enumeration of the storage system is completed.

[0072] It is understandable that in this step, the processor can determine that all data in the storage system have been read and enumerated after completing the reading of the target unlisted data and determining that there is no other unlisted data in the current enumerated fragment sequence.

[0073] In this embodiment, the embodiment of the present invention reads the target unlisted data from the target shard according to the current enumerated shard sequence, and can arrange the data in an orderly manner during the data reading process of the storage system, which is convenient for customers to use. There is no need to sort and organize the massive data after reading it out. Therefore, in the case of massive data, the storage system can be enumerated in an orderly manner with less memory overhead and computing overhead, thereby improving the user experience.

[0074] Based on the above embodiment, the present invention also provides another method for orderly listing data in a storage system to correct the storage location of data stored in each shard in the storage system and improve the subsequent listing speed. Figure 3 , Figure 3 This is a flowchart of another method for orderly enumerating data in a storage system provided by an embodiment of the present invention. The method may include:

[0075] Step 201: After obtaining the data enumeration instruction of the storage system, the current shard sequence of the storage system is determined as the current enumeration shard sequence; wherein the current shard sequence is a shard sequence sorted from small to large according to the smallest data in the shard, and the data in the shard is sorted from small to large.

[0076] It can be understood that the data enumeration instruction in this embodiment can be an instruction for effectively enumerating the data in all shards in the storage system. The current shard sequence is a shard sequence that is currently sorted from small to large according to the smallest data in the shard, that is, the smallest data (i.e., the smallest data) in the first shard in the current shard sequence is the smallest data in the storage system, and the minimum data of the previous shard in the current shard sequence is less than the minimum data in the next shard. The data in the shards are sorted from small to large, that is, the first data in each shard is the smallest data (i.e., the smallest data) in the shard, and the previous data in the shard is less than the next data, that is, the first data to the last data in each shard increases in sequence.

[0077] Correspondingly, in this step, after the processor obtains the data enumeration instruction of the storage system, since the data in the current shard sequence are all read enumeration, the current shard sequence can be directly determined as the current enumeration shard sequence, that is, the data in each shard in the current shard sequence are all determined as unenumerated data.

[0078] It should be noted that the method provided in this embodiment may also include a process for generating and updating a current shard sequence. For example, after the data in the current shard changes, the processor may sort the shards according to the size of the first data (i.e., the smallest data) in each shard in the storage system, and update the current shard sequence; wherein the current shard is any shard in the storage system. For example, after inserting data into the shards according to the data size, the processor may sort the first data in each shard according to the size of the data, and update the current shard sequence.

[0079] For example, if Figure 4 As shown in the figure, the first shard (shard-1) is filled with data. After the new shard is written with data, the new shard is compared with the first data (i.e., the minimum data) of the original shard. Figure 5 As shown in , since the first data of the new shard (a-file) is smaller than the first data of the original shard (e-file), the new shard is marked as the first shard (shard-1) of the current shard sequence, and the original shard is marked as the second shard (shard-2) of the current shard sequence; Figure 6 As shown, after the third shard writes data, the first data of the shard (ie, the minimum data) is compared with other shards, and the shard number of the current shard sequence is updated.

[0080] Step 202: Find the position of the positioning data in the target slice; wherein the target slice is the first slice in the current listed slice sequence, the positioning data is the first unlisted data in the data sorting of the target slice that is greater than the comparison data, and the comparison data is the smallest unlisted data in the second slice in the current listed slice sequence.

[0081] It is understandable that in this embodiment, the processor may start reading data from the first slice (i.e., the target slice) in the currently listed slice sequence each time it reads data, and find a position in the first slice that is larger than the first data in the second slice (i.e., the comparison data); Figure 6 As shown, the position of d-file (i.e., positioning data) that is larger than b-file in shard-2 is found in shard-1 to read the unlisted data (a-file) before d-file in shard-1.

[0082] Correspondingly, this embodiment does not limit the specific method in which the processor searches for the position of the positioning data in the target slice. For example, the processor may use a binary search method to quickly search for the position of the positioning data in the target slice.

[0083] Step 203: Determine whether the located data is smaller than the smallest unlisted data in the third fragment in the currently listed fragment sequence; if so, proceed to step 204; if not, proceed to step 205.

[0084] Among them, in this step, the processor can determine whether the data stored on the slice in the current storage system needs to adjust the storage position by judging whether the positioning data in the target slice is smaller than the smallest unlisted data in the third slice in the currently listed slice sequence. Therefore, when the positioning data is smaller than the smallest unlisted data in the third slice in the currently listed slice sequence, the storage position of the positioning data in the target slice and the comparison data in the second slice in the currently listed slice sequence are swapped through step 204, so that the data on the slice can be continuously ordered to improve the enumeration speed of the next orderly enumeration of data.

[0085] Step 204: swap the storage locations of the positioning data and the comparison data.

[0086] Specifically, in this step, the processor can swap the storage positions of the positioning data in the first slice (i.e., the target slice) and the comparison data in the second slice when the positioning data in the first slice (i.e., the target slice) in the currently enumerated slice sequence is greater than the comparison data in the second slice and is less than the smallest unlisted data in the third slice, so that the data on the slices in the storage system can be continuously ordered.

[0087] For example, if you encounter Figure 7In the case shown, data is read from shard-1. After comparing the first data of shard-1 with that of shard-2, the location of shard-1's e-file (i.e., the location data) is found. Since the e-file is smaller than the first data of shard-3 (f-file), the positions of shard-1's e-file and shard-2's first data (d-file) can be swapped.

[0088] Step 205: Determine the unlisted data before the location of the located data in the target slice as target unlisted data, and read the target unlisted data.

[0089] It is understandable that in this step, the processor can determine the unlisted data in the target slice that is smaller than the comparison data as the target unlisted data, and read the target unlisted data, such as reading the target unlisted data in sequence according to the sorting order in the target slice.

[0090] Correspondingly, in this step, after the processor reads the target unlisted data, it may also determine the target unlisted data as listed data.

[0091] For example, if Figure 6 As shown, after the processor finds the location of d-file (i.e., the positioning data) in shard-1 that is larger than b-file in shard-2, it can read the unlisted data (a-file) before d-file in shard-1.

[0092] Step 206: If there is unlisted data in the current enumerated fragment sequence, the current enumerated fragment sequence is updated according to the unlisted data in the current enumerated fragment sequence, and the process proceeds to step 202.

[0093] In this step, after the target unlisted data is read, if the processor determines that there is other unlisted data in the current listed fragment sequence, it can sort the fragments according to the size of the smallest unlisted data in the fragments, update the current listed fragment sequence, and return to step 202 to continue reading the target unlisted data in the updated current listed fragment sequence.

[0094] For example, if Figure 6As shown in the figure, after the processor finds the location of d-file (i.e., the location data) in shard-1 that is larger than b-file in shard-2, it can read the unlisted data (a-file) before d-file in shard-1. Since the first unlisted data (i.e., the smallest unlisted data, d-file) in shard-1 is larger than b-file in shard-2 and smaller than e-file in shard-3, the current shard enumeration sequence can be adjusted and updated, as shown in the figure. Figure 7 As shown, the original shard-2 is adjusted to shard-1 of the current shard sequence, and the original shard-1 is adjusted to shard-2 of the current shard sequence.

[0095] Step 207: If there is no unlisted data in the current enumerated fragment sequence, it is determined that the data enumeration of the storage system is completed.

[0096] It is understandable that in this step, the processor can determine that all data in the storage system have been read and enumerated after completing the reading of the target unlisted data and determining that there is no other unlisted data in the current enumerated fragment sequence.

[0097] In this embodiment, the embodiment of the present invention swaps the storage positions of the positioning data in the first slice and the comparison data in the second slice when the positioning data in the first slice in the current enumerated slice sequence is greater than the comparison data in the second slice and is less than the smallest unenumerated data in the third slice, so that the data on the slices in the storage system can be continuously ordered, corrects the storage position of the data stored in each slice in the storage system, and improves the enumeration speed of the next orderly enumeration of data.

[0098] Corresponding to the above method embodiment, an embodiment of the present invention further provides an orderly enumeration data device for a storage system. The orderly enumeration data device for a storage system described below and the orderly enumeration data method for a storage system described above can refer to each other.

[0099] Please refer to Figure 8 , Figure 8 This is a block diagram of a device for orderly enumerating data in a storage system provided by an embodiment of the present invention. The device may include:

[0100] The sequence acquisition module 10 is used to obtain the current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shard, and the data in the shard is sorted by data size;

[0101] The data reading module 20 is configured to read target unlisted data from a target shard according to the currently listed shard sequence; wherein the target shard is the first shard or the last shard in the currently listed shard sequence; the target unlisted data is unlisted data in the target shard that is smaller than unlisted data in other shards or unlisted data in the target shard that is larger than unlisted data in other shards; and unlisted data is data that has not been read from the list.

[0102] A sequence updating module 30 is configured to update the current enumerated fragment sequence according to the unenumerated data in the current enumerated fragment sequence if there is unenumerated data in the current enumerated fragment sequence, and send a start signal to the data reading module;

[0103] The completion determination module 40 is configured to determine that the data enumeration of the storage system is completed if there is no unenumerated data in the current enumerated fragment sequence.

[0104] In some embodiments, the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data in the shard, and the target unenumerated data is the unenumerated data in the target shard that is smaller than the unenumerated data in other shards.

[0105] In some embodiments, the current enumerated fragment sequence is a fragment sequence sorted from smallest to largest according to the smallest unenumerated data in the fragment, and the data reading module 20 may include:

[0106] The shard determination submodule is used to determine the first shard in the currently enumerated shard sequence as the target shard;

[0107] The data reading submodule is used to read the target unlisted data from the target shard; wherein the target unlisted data is the unlisted data in the target shard that is smaller than the comparison data, and the comparison data is the smallest unlisted data in the second shard in the current listed shard sequence.

[0108] In some embodiments, the data in the shards are sorted from small to large, and the data reading submodule may include:

[0109] A positioning unit, configured to locate the location of positioning data in the target shard; wherein the positioning data is the first unlisted data in the data sorting of the target shard that is greater than the comparison data;

[0110] The reading unit is configured to determine the unlisted data before the location of the positioning data in the target slice as target unlisted data, and read the target unlisted data.

[0111] In some embodiments, the data reading submodule may further include:

[0112] a judgment unit, configured to judge whether the located data is smaller than the smallest unlisted data in the third fragment in the currently listed fragment sequence; and if the located data is smaller than the smallest unlisted data in the third fragment in the currently listed fragment sequence, sending a start signal to the reading module;

[0113] a correction unit, configured to swap the storage locations of the positioning data and the comparison data if the positioning data is smaller than the smallest unlisted data in the third slice in the currently listed slice sequence, and send a start signal to the reading module;

[0114] In some embodiments, the positioning unit may be specifically configured to use a binary search method to find the position of the positioning data in the target slice.

[0115] In some embodiments, the sequence acquisition module 10 may be specifically configured to, after acquiring a data enumeration instruction of the storage system, determine the current shard sequence of the storage system as the current enumerated shard sequence;

[0116] Correspondingly, the device may further include:

[0117] The storage update module is used to sort the shards according to the size of the minimum or maximum data in each shard in the storage system and update the current shard sequence after the data in the current shard changes; wherein the current shard is any shard.

[0118] In this embodiment, the embodiment of the present invention reads the target unlisted data from the target shard according to the current enumerated shard sequence through the data reading module 20, and can arrange the data in an orderly manner during the data reading process of the storage system, which is convenient for customers to use. There is no need to sort and organize the massive data after reading it out. Therefore, in the case of massive data, the storage system can be enumerated in order with less memory overhead and computing overhead, thereby improving the user experience.

[0119] Corresponding to the above method embodiment, an embodiment of the present invention further provides an ordered data enumeration device for a storage system. The ordered data enumeration device for a storage system described below and the ordered data enumeration method for a storage system described above can refer to each other.

[0120] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of a device for orderly enumerating data in a storage system provided by an embodiment of the present invention. The device for orderly enumerating data may include:

[0121] Memory D1, for storing computer programs;

[0122] The processor D2 is configured to implement the steps of the method for orderly enumerating data in the storage system provided by the above method embodiment when executing a computer program.

[0123] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of the specific structure of an ordered enumerated data device of a storage system provided in an embodiment of the present invention. The ordered enumerated data device 310 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) for storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the ordered enumerated data device 310.

[0124] The ordered enumerated data device 310 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0125] Specifically, the device for orderly enumerating data of the storage system provided in this embodiment may be a server, such as a server of the storage system.

[0126] The steps in the method for orderly enumerating data of a storage system described above can be implemented by the structure of a device for orderly enumerating data of the storage system.

[0127] Corresponding to the above method embodiment, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium described below and the method for orderly enumerating data in a storage system described above can refer to each other.

[0128] Please refer to Figure 11 , Figure 11 The computer-readable storage medium 50 stores a computer program 51, which, when executed by a processor, implements the steps of the method for sequentially enumerating data in a storage system as provided in the above method embodiment.

[0129] The computer-readable storage medium 50 may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other storage media capable of storing program codes.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. References to the common and similar parts between the various embodiments are sufficient. The devices, apparatuses, and computer-readable storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the description of the methods.

[0131] The above is a detailed introduction to the method, device, equipment and computer-readable storage medium for orderly enumerating data of a storage system provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for orderly enumerating data in a storage system, characterized in that: include: Obtaining a current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shard, and the data in the shard is sorted by data size; Read target unlisted data from a target shard according to the currently listed shard sequence; wherein the target shard is the first shard or the last shard in the currently listed shard sequence; the target unlisted data is unlisted data in the target shard that is smaller than unlisted data in other shards or unlisted data in the target shard that is larger than unlisted data in other shards, and the unlisted data is data that has not been enumerated and read; If there is unlisted data in the currently listed fragment sequence, the currently listed fragment sequence is updated according to the unlisted data in the currently listed fragment sequence, and the step of reading the target unlisted data from the target fragment according to the currently listed fragment sequence is performed; wherein the updating of the currently listed fragment sequence includes: when the positioning data in the target fragment is smaller than the smallest unlisted data in the third fragment in the currently listed fragment sequence, swapping the storage positions of the positioning data and the comparison data, wherein the positioning data is the first unlisted data in the data sorting of the target fragment that is larger than the comparison data, and the comparison data is the smallest unlisted data in the second fragment in the currently listed fragment sequence; If there is no unlisted data in the current enumerated fragment sequence, it is determined that the data enumeration of the storage system is completed.

2. The method for orderly enumerating data in a storage system according to claim 1, characterized in that: The current enumerated fragment sequence is a fragment sequence sorted by the size of the smallest unenumerated data in the fragment, and the target unenumerated data is the unenumerated data in the target fragment that is smaller than the unenumerated data in other fragments.

3. The method for orderly enumerating data in a storage system according to claim 2, wherein: The current enumerated shard sequence is a shard sequence sorted from smallest to largest according to the smallest unenumerated data in the shard. Reading the target unenumerated data from the target shard according to the current enumerated shard sequence includes: Determine the first fragment in the currently enumerated fragment sequence as the target fragment; The target unlisted data is read from the target slice; wherein the target unlisted data is unlisted data in the target slice that is smaller than comparison data, and the comparison data is the smallest unlisted data in the second slice in the current listed slice sequence.

4. The method for orderly enumerating data in a storage system according to claim 3, characterized in that: The data in the shards are sorted from small to large, and the reading of the target unlisted data from the target shards includes: Find the position of the positioning data in the target slice; wherein the positioning data is the first unlisted data in the data sorting of the target slice that is greater than the comparison data; The unlisted data before the position of the positioning data in the target slice is determined as the target unlisted data, and the target unlisted data is read.

5. The method for orderly enumerating data in a storage system according to claim 4, characterized in that: After searching for the location of the positioning data in the target slice, the method further includes: Determine whether the positioning data is smaller than the smallest unlisted data in the third fragment in the currently listed fragment sequence; If so, swapping the storage locations of the positioning data and the comparison data, and executing the steps of determining the unlisted data before the location of the positioning data in the target slice as the target unlisted data, and reading the target unlisted data; If not, the steps of determining the unlisted data before the position of the positioning data in the target slice as the target unlisted data and reading the target unlisted data are performed.

6. The method for orderly enumerating data in a storage system according to claim 4, characterized in that: The step of searching for the location of the positioning data in the target slice includes: The position of the positioning data in the target slice is found using a binary search method.

7. The method for orderly enumerating data in a storage system according to any one of claims 1 to 6, characterized in that: Obtaining the current enumerated shard sequence of the storage system includes: After obtaining the data enumeration instruction of the storage system, determining the current shard sequence of the storage system as the current enumeration shard sequence; Correspondingly, the method for orderly enumerating data further includes: After the data in the current shard changes, the shards are sorted according to the size of the minimum data or the maximum data in each shard in the storage system, and the current shard sequence is updated; wherein the current shard is any of the shards.

8. A device for orderly enumerating data in a storage system, characterized in that: include: A sequence acquisition module is configured to acquire a current enumerated shard sequence of the storage system; wherein the current enumerated shard sequence is a shard sequence sorted by the size of the smallest unenumerated data or the largest unenumerated data in the shard, and the data in the shard is sorted by data size; A data reading module is configured to read target unlisted data from a target shard according to a currently listed shard sequence; wherein the target shard is the first shard or the last shard in the currently listed shard sequence; the target unlisted data is unlisted data in the target shard that is smaller than unlisted data in other shards or unlisted data in the target shard that is larger than unlisted data in other shards, and the unlisted data is data that has not been enumerated and read; a sequence updating module configured to update the current enumerated fragment sequence according to the unlisted data in the current enumerated fragment sequence if there is unlisted data in the current enumerated fragment sequence, and to send a start signal to the data reading module; wherein the updating of the current enumerated fragment sequence includes: when the positioning data in the target fragment is less than the smallest unlisted data in the third fragment in the current enumerated fragment sequence, swapping the storage positions of the positioning data and the comparison data, wherein the positioning data is the first unlisted data in the data sorting of the target fragment that is larger than the comparison data, and the comparison data is the smallest unlisted data in the second fragment in the current enumerated fragment sequence; The completion determination module is used to determine that the data enumeration of the storage system is completed if there is no unenumerated data in the current enumerated fragment sequence.

9. A device for enumerating data in an ordered manner in a storage system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for orderly enumerating data in a storage system as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for orderly enumerating data in a storage system according to any one of claims 1 to 7.

Images