Data traversal method and device of storage system, electronic equipment and storage medium

Abstract

The application provides a data traversal method and device of a storage system, an electronic device and a storage medium, relates to the technical field of data storage, and through a conversion function, a converted identifier set has a more uniform distribution characteristic than an original object identifier set under a predetermined ordering rule, and based on the uniform distribution, the converted identifier set can be divided into multiple continuous and non-overlapping identifier intervals. Each interval can call a list interface of a distributed object storage system in parallel through an independent traversal subtask to carry out traversal, and finally, the original object identifier can be obtained through an inverse function of the conversion function. Therefore, the problems of hotspot prefix bottleneck caused by uneven distribution of original object identifiers and low efficiency caused by serial execution limitation in the prior art can be solved, and the technical effects of significantly improving the traversal efficiency of massive objects, achieving load balancing in traversal, and accurately restoring the original object identifier are achieved.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data traversal method and apparatus, electronic device and storage medium for a storage system. Background Technology

[0002] Object storage, with its advantages of high scalability, high availability, and low cost, is widely used in scenarios such as data backup, data lakes, and cloud-native applications. These storage systems employ a key-value pair storage model, distinguishing each object by a unique object identifier, rather than the directory structure of a traditional file system. This allows for efficient management of massive amounts of data, but also presents challenges in traversal efficiency. The main traversal interface of object storage returns a limited number of objects per call, and subsequent calls must depend on the return identifier of the previous call. It can only be executed sequentially, preventing multiple simultaneous calls. When dealing with massive amounts of objects, numerous consecutive calls are required, resulting in a lengthy traversal process that struggles to meet the demands of real-time data processing and rapid migration scenarios.

[0003] In practice, many object identifiers tend to cluster under a common prefix, with the number of objects under this prefix far exceeding those under other prefixes. This makes traversing this prefix a bottleneck, further lengthening the overall traversal time and causing uneven load distribution. Previous optimization attempts were either limited by the interface order, making true parallel traversal difficult, or had limited optimization effects when object identifiers lacked a clear prefix pattern or exhibited the aforementioned clustered prefixes. Summary of the Invention

[0004] This application provides a data traversal method and apparatus, electronic device and storage medium for a storage system, to at least partially solve one of the technical problems in the related art.

[0005] This application provides a data traversal method for a storage system, including:

[0006] Obtain the original object identifier, generate a corresponding transformed identifier for each original object identifier through a transformation function and store it. The transformation function is configured to make the set of transformed identifiers have a more uniform distribution characteristic than the set of original object identifiers under a predetermined sorting rule. Based on the sorting rules of the converted identifiers, the set of converted identifiers is divided into multiple consecutive and non-overlapping identifier intervals. For each identifier range, a separate traversal subtask is started. In each traversal subtask, the list interface of the distributed object storage system is called to traverse the range based on the boundary information of the identifier range. Collect the traversal results of all traversal subtasks, and for each transformed identifier in the traversal results, apply the inverse function of the transformation function to restore the corresponding original object identifier.

[0007] This application also provides a data traversal device for a storage system, comprising: The generation unit is used to obtain the original object identifier, generate a corresponding transformed identifier for each original object identifier through a transformation function and store it. The transformation function is configured to make the set of transformed identifiers have a more uniform distribution characteristic than the set of original object identifiers under a predetermined sorting rule. The partitioning unit is used to divide the set of converted identifiers into multiple consecutive and non-overlapping identifier intervals according to the sorting rules of the converted identifiers. The traversal unit is used to start an independent traversal subtask for each identifier range. In each traversal subtask, the list interface of the distributed object storage system is called to traverse based on the boundary information of the identifier range. The restoration unit is used to collect the traversal results of all traversal subtasks. For each transformed identifier in the traversal result, the inverse function of the transformation function is applied to restore the corresponding original object identifier.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above methods.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0011] This application provides a data traversal method, apparatus, electronic device, and storage medium for a storage system. By using a conversion function, the converted set of identifiers exhibits a more uniform distribution than the original set of object identifiers under a predetermined sorting rule. Based on this uniform distribution, the converted set of identifiers can be divided into multiple continuous, non-overlapping identifier intervals. Each interval can be traversed in parallel by calling the list interface of the distributed object storage system through independent traversal subtasks. Finally, the original object identifiers can be restored using the inverse function of the conversion function. Therefore, this application solves the problems in existing technologies where uneven distribution of original object identifiers leads to hotspot prefix bottlenecks and low efficiency due to serial execution limitations during traversal. It achieves the technical effects of significantly improving the efficiency of traversing massive objects, realizing load balancing during traversal, and accurately restoring the original object identifiers.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart illustrating a data traversal method for a storage system provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data traversal device for a storage system provided in an embodiment of this application. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The specific application environment architecture or specific hardware architecture on which the execution of the data traversal method of the storage system depends is described here.

[0017] Embodiments of this application provide a data traversal method for a storage system. Figure 1 This is a flowchart illustrating a data traversal method for a storage system provided in an embodiment of this application.

[0018] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the original object identifiers, generate corresponding converted identifiers for each original object identifier using a conversion function, and store them. The conversion function is configured to make the set of converted identifiers have a more uniform distribution characteristic than the set of original object identifiers under a predetermined sorting rule.

[0019] In this embodiment, the original object identifiers (i.e., ObjectKeys) in the distributed object storage system are obtained. Each original object identifier is processed by a preset conversion function to generate a corresponding converted identifier, which is then stored. The core design goal of this conversion function is to optimize the distribution characteristics of the identifiers, ensuring that the set of converted identifiers exhibits a more uniform distribution than the set of original object identifiers under the predetermined sorting rules followed by the distributed object storage system. This avoids the local density problem caused by the concentrated distribution of the original object identifiers. The conversion function can achieve the above distribution optimization through mapping, encoding, or other functional processing methods. Its specific implementation is not limited, as long as the core effect of uniform distribution is achieved. As one implementation method, the conversion function can use a hash algorithm to calculate the original object identifiers, map the calculation result to a preset identifier prefix, and then connect the prefix to the original object identifiers through a separator to form the converted identifiers, thus conforming to the lexicographical sorting rules of the distributed object storage system and achieving a balanced distribution of the converted identifiers.

[0020] By optimizing the distribution characteristics of identifiers through the transformation function, the hotspot problem caused by the uneven distribution of original object identifiers is alleviated from the root. This provides the foundation for subsequent partitioned parallel traversal based on the transformed identifiers and helps to improve the load balance and efficiency of the overall traversal process.

[0021] Step 102: According to the sorting rules of the converted identifiers, the set of converted identifiers is divided into multiple consecutive and non-overlapping identifier intervals.

[0022] In this embodiment, based on a predetermined sorting rule followed by the converted identifiers, the complete set of converted identifiers is divided into multiple continuous and non-overlapping identifier intervals. Each interval collectively covers all converted identifiers, ensuring that no identifier is omitted or duplicated, providing a clear and orderly basis for subsequent parallel traversal. As one implementation, when the predetermined sorting rule is lexicographical, the interval division boundaries can be determined by combining the distribution characteristics of the converted identifiers, so that each interval corresponds to a set of consecutively sorted converted identifiers, adapting to the list interface call logic of the distributed object storage system.

[0023] The interval partitioning method is deeply adapted to the sorting rules of the converted identifiers. The continuous and non-overlapping design ensures the complete coverage of the identifier set, which not only clears the conflict obstacles for subsequent parallel traversal of multiple subtasks, but also ensures the accuracy of the traversal range, laying the foundation for improving the overall traversal efficiency.

[0024] Step 103: For each identifier range, start an independent traversal subtask. In each traversal subtask, call the list interface of the distributed object storage system to traverse based on the boundary information of the identifier range.

[0025] In this embodiment, for each consecutive and non-overlapping identifier interval obtained by the aforementioned division, an independent traversal subtask is initiated. These subtasks can be executed in parallel without interference. Each subtask, based on the boundary information of its identifier interval, clarifies the start and end ranges of the traversal, and then calls the list interface (i.e., ListObjectsV2 API) of the distributed object storage system to perform targeted traversal, ensuring that only the converted identifiers within the specified interval are traversed, avoiding overlapping or omissions in the traversal range. As one implementation method, the traversal subtasks can rely on a thread pool for resource scheduling. When calling the list interface, the identifier filtering parameters and start position parameters corresponding to the interval are configured. Combined with the pagination identifier returned by the interface, the complete traversal within the interval is completed. Furthermore, caching the traversal progress allows for resuming traversal after a failure.

[0026] By assigning independent and parallel traversal subtasks to each identifier range, the limitations of traditional list interface serial calls are broken, significantly improving the traversal efficiency of massive objects. Relying on range boundary information to accurately locate the traversal range ensures the accuracy and completeness of the traversal, while achieving load balancing of each traversal subtask and preventing a single range from becoming a traversal bottleneck.

[0027] Step 104: Gather the traversal results of all traversal subtasks, and apply the inverse function of the transformation function to each transformed identifier in the traversal results to restore the corresponding original object identifier.

[0028] In this embodiment, after all independent traversal subtasks are completed, the traversal results output by each subtask are aggregated to ensure that all transformed identifiers are included in the result set without omission or duplication. Then, for each transformed identifier in the set, the inverse function corresponding to the aforementioned transformation function is applied for processing. Through the reverse mapping function, the original object identifier (i.e., Object Key) corresponding to each transformed identifier is accurately restored, and the restoration process does not change the inherent attributes or information integrity of the original object identifier. As one implementation, when the transformation function adds a uniformly distributed prefix, the inverse function can be expressed as stripping the prefix and preset separator from the transformed identifier to restore the original form of the original object identifier.

[0029] This processing method ensures the integrity of object traversal by uniformly aggregating the traversal results of each parallel subtask. At the same time, it uses the inverse function of the transformation function to accurately restore the original object identifier. This not only does not affect the business side's need to use the original data, but also works in synergy with the aforementioned distributed optimization and parallel traversal techniques to ensure the unity of efficient traversal and data accuracy.

[0030] This application provides a data traversal method for a storage system. By using a transformation function, the set of transformed identifiers has a more uniform distribution characteristic than the original set of object identifiers under a predetermined sorting rule. Based on this uniform distribution, the set of transformed identifiers can be divided into multiple continuous and non-overlapping identifier intervals. Each interval can be traversed in parallel by calling the list interface of the distributed object storage system through independent traversal subtasks. Finally, the original object identifiers can be restored through the inverse function of the transformation function. Therefore, it can solve the problems of hotspot prefix bottleneck caused by uneven distribution of original object identifiers and low efficiency caused by serial execution limitation in the existing technology. It achieves the technical effects of significantly improving the efficiency of massive object traversal, realizing traversal load balancing, and accurately restoring the original object identifiers.

[0031] In the embodiments involved in this application, there are various feasible specific implementation methods. To clearly and completely illustrate the technical solutions of this disclosure, the implementation methods listed below are merely exemplary and do not constitute a limitation on the scope of protection of this disclosure. That is, in addition to the implementation methods described below, other implementation methods that can be obtained by those skilled in the art based on the technical content disclosed in this disclosure through reasonable logical analysis, reasoning, or limited experimentation should also be covered within the scope of protection of this disclosure. The following specifically describes some exemplary implementation methods: As a specific implementation of this application, based on the basic scheme, the conversion function is further defined as a hash function; the hash function is configured to: receive the byte sequence of the original object identifier as input; generate the corresponding hash value; select a character from a predetermined ordered character set as a prefix according to the hash value; and concatenate the prefix with the original object identifier to form the converted identifier.

[0032] Specifically, the implementation details are as follows: In this embodiment, the conversion function is a hash function (a non-cryptographic hash function). The execution logic of this hash function is as follows: It receives the UTF-8 encoded byte sequence of the original object identifier (i.e., Object Key) as input data and generates a 32-bit or 64-bit hash value through hash operation; the predetermined ordered character set is set as a lowercase letter set (az). Based on the number of characters N in this character set (default N=26, but can be adjusted to the number corresponding to a specified letter range according to business needs), a modulo operation is performed on the generated hash value, and the character corresponding to the modulo result is the candidate prefix; a preset separator (such as "_", which is a character that is not commonly used in the original object identifier to avoid conflicts) is used to connect the prefix and the original object identifier, and finally the converted identifier is formed by concatenation.

[0033] By combining non-encrypted hash functions with prefix selection based on ordered character sets, it is ensured that the converted identifiers are evenly distributed under the predetermined sorting rules, effectively avoiding the problem of hotspot prefixes formed by concentrated original object identifiers. The setting of the separator not only ensures a clear distinction between the prefix and the original object identifier, but also provides a clear identification basis for subsequent inverse function restoration operations, further guaranteeing the accuracy and efficiency of traversal and restoration.

[0034] As a specific implementation of this application, based on the basic scheme, multiple independent traversal subtasks are further limited to be executed in parallel.

[0035] Specifically, multiple independent traversal subtasks are executed in parallel through a thread pool. The initial number of threads in the thread pool is consistent with the number of identifier intervals defined above. Each traversal subtask corresponds to a continuous and non-overlapping identifier interval and is bound to an independent thread resource. Before each subtask starts, the access credentials of the distributed object storage system, the target bucket name, the prefix parameter (i.e., the prefix parameter) corresponding to the identifier interval, and the initial start position parameter (i.e., the start-after parameter) are configured. At the same time, the local cache module is initialized to record the transformed identifiers that have been traversed and the latest page identifier (i.e., the ContinuationToken). The system manages all threads uniformly through a collaborative scheduling module. Each subtask independently calls the list interface (i.e., ListObjectsV2 API) of the distributed object storage system to perform traversal. The traversal range is filtered based on the boundary information of the interval to which it belongs, and only the converted identifiers within the interval are processed, without interference between them. Furthermore, the collaborative scheduling module monitors for faults by periodically checking thread heartbeats (e.g., once every 10 seconds). If the thread corresponding to a subtask does not respond for more than a preset time, it is determined to be a fault, and the system automatically restarts the subtask with the cached last traversal position as the new starting parameter to ensure the continuity of parallel traversal.

[0036] By using thread pool scheduling to achieve parallel execution of multiple traversal subtasks, the limitation of traditional serial calls to list interfaces is completely broken, making the traversal time approximately inversely proportional to the number of intervals, and greatly improving the traversal efficiency of massive objects. Each subtask only processes the identifier of the corresponding interval. Combined with the evenly distributed transformed identifiers, the traversal load is evenly distributed, effectively avoiding the single-thread bottleneck caused by hotspot prefixes, while ensuring the integrity and accuracy of the traversal.

[0037] As a specific implementation of this application, based on the basic solution, this application embodiment further includes, after restoring the corresponding original object identifier, sorting the original object identifier according to a predetermined sorting rule.

[0038] Specifically, after restoring all corresponding original object identifiers (i.e., Object Keys) through the inverse function of the transformation function, all original object identifiers are sorted according to a preset sorting rule. This preset sorting rule can be consistent with the sorting rules followed by the distributed object storage system (such as UTF-8 lexicographical order). The sorting process makes full use of the inherent ordered characteristics of the results returned by each traversal subtask—since the identifiers after transformation are divided into intervals and traversed according to the preset rule, the original object identifiers corresponding to each interval still maintain the ordered state within the interval after restoration. Therefore, the merge sort algorithm is used to integrate the original object identifiers of all intervals. There is no need to perform overall unordered sorting of all data. A list of overall ordered original object identifiers can be quickly obtained by merging multiple ordered subsets. The sorting process does not change any information of the original object identifiers, but only adjusts the output order to conform to the preset rule.

[0039] This sorting method precisely matches the business requirement for the ordered output of the original object identifiers. By leveraging the merge sort algorithm to fully reuse the ordered characteristics of each interval, it significantly reduces the amount of sorting computation while ensuring sorting accuracy and avoiding additional performance loss. It works in synergy with the aforementioned parallel traversal and accurate restoration techniques to further enhance the practicality and adaptability of the technical solution.

[0040] As a specific implementation of this application, based on the basic scheme, the inverse function of the conversion function is further configured to remove a fixed-length prefix from the converted identifier.

[0041] Specifically, the inverse function of the conversion function is configured to perform a fixed-length prefix removal operation on the converted identifier. This fixed length is pre-defined to match the length of the prefix portion added when the conversion function generates the converted identifier. When generating the converted identifier, the conversion function has already specified that the total length of the prefix portion (including the pre-defined prefix character and separator) is a fixed value (e.g., the prefix is ​​one lowercase letter, the separator is one pre-defined character, and the overall fixed length is 2). The inverse function does not require complex parsing logic; it only needs to extract the pre-defined fixed-length character segment from the beginning of the converted identifier and remove it directly. The remaining character sequence is the corresponding original object identifier (i.e., the Object Key). For example, in the converted identifier "m_data / file1.txt", the fixed-length prefix portion is "m_" (length 2). After removing this portion, the inverse function can directly restore the original object identifier "data / file1.txt".

[0042] The implementation of this inverse function is simple to operate and highly efficient. It requires no additional recognition or calculation process and can quickly restore the original object identifier, reducing system performance loss. At the same time, based on the fixed-length prefix removal logic, it avoids restoration errors caused by identifier format differences, further ensuring the accuracy and reliability of the restoration results.

[0043] As a specific implementation of this application, based on the basic scheme, the boundary information of the identifier interval is further defined by writing the marker objects corresponding to the boundaries of each interval to the distributed object storage system.

[0044] Specifically, the boundary information of the identifier intervals is determined by writing the corresponding tag objects to the distributed object storage system (i.e., the S3 object storage system). After the set of identifiers is divided into multiple continuous and non-overlapping identifier intervals after the transformation, a unique tag object is generated for each interval. The identifier of this tag object is configured according to a predetermined sorting rule to be exactly located at the starting boundary position of the corresponding interval, and its naming format is consistent with the transformed identifier to ensure sorting adaptation (for example, when the transformed identifier is composed of a letter prefix + separator + original object identifier, the identifier of the tag object can be set to the prefix character of the corresponding interval itself). After these tag objects are written to the distributed object storage system one by one, each tag object will naturally locate the boundary of the corresponding interval based on the system's predetermined sorting rule. When the traversal subtask starts, the starting boundary information of the interval to which it belongs can be determined by reading the identifier of the tag object. Combined with the tag object corresponding to the ending boundary of the interval (or the starting tag object of the next interval), the traversal range is accurately determined, ensuring that each interval is continuous and non-overlapping.

[0045] By writing the marker object to determine the boundary information of the identifier range, the sorting characteristics of the distributed object storage system can be relied upon to achieve accurate boundary positioning and avoid deviations in the range division. At the same time, it provides an intuitive and reliable boundary basis for traversing subtasks, simplifies the subtask's judgment logic on the traversal range, further ensures the accuracy and completeness of parallel traversal, and avoids the situation of identifier omission or repeated traversal.

[0046] As a specific implementation of this application, based on the basic scheme, it is further specified that when a failure is detected in the traversal subtask, the failed traversal subtask is automatically restarted, and the traversal continues from the last successfully traversed identifier before the failure of the subtask.

[0047] Specifically, the system is configured with a collaborative scheduling module and a local caching unit to implement fault monitoring and breakpoint resumption of traversal subtasks. The collaborative scheduling module sends heartbeat detection signals to each independent traversal subtask at a preset period (e.g., once every 10 seconds) to monitor the running status of the subtasks in real time. If a traversal subtask fails to return a heartbeat response for more than a preset timeout threshold (e.g., 30 seconds), it is determined that the subtask has failed. During the execution of each traversal subtask, the local caching unit continuously records the converted identifiers (i.e., the prefixed Object Keys) of successfully traversed subtasks and the latest paging identifier (i.e., ContinuationToken). This cached information corresponds one-to-one with the identifier range to which the subtask belongs and is updated in real time to the state of the last successful traversal before the failure. When a failed subtask is automatically restarted, the system reads the last successfully traversed transformed identifier recorded in its local cache before the failure and uses this identifier as the starting position parameter (i.e., the start-after parameter) for the traversal after the restart. At the same time, the system keeps the configuration of the identifier interval boundary information, target prefix parameter and other configurations of the subtask consistent with those before the failure. The restarted subtask calls the list interface (i.e., ListObjectsV2 API) of the distributed object storage system and continues to execute the traversal from this starting position until the traversal of the entire interval is completed.

[0048] This fault handling mechanism achieves rapid fault identification through heartbeat monitoring and accurately records traversal progress with the help of local caching, ensuring seamless continuation from the breakpoint after restarting and avoiding repeated processing of already traversed objects. This not only guarantees the integrity and efficiency of traversal but also improves the stability and fault tolerance of the entire traversal system.

[0049] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0050] Corresponding to the data traversal method for the storage system described above, this disclosure also proposes a data traversal apparatus for the storage system. Since the apparatus embodiments of this disclosure correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to the method embodiments described above, and will not be repeated here.

[0051] Figure 2 This is a schematic diagram of the structure of a data traversal device for a storage system provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, it includes: The generation unit 21 is used to obtain the original object identifier, generate a corresponding converted identifier for each original object identifier through a conversion function and store it. The conversion function is configured to make the set of converted identifiers have a more uniform distribution characteristic than the set of original object identifiers under a predetermined sorting rule. The partitioning unit 22 is used to divide the set of converted identifiers into multiple continuous and non-overlapping identifier intervals according to the sorting rules of the converted identifiers. Traversal unit 23 is used to start an independent traversal subtask for each identifier range. In each traversal subtask, the list interface of the distributed object storage system is called to traverse based on the boundary information of the identifier range. The restoration unit 24 is used to collect the traversal results of all traversal subtasks. For each transformed identifier in the traversal result, the inverse function of the transformation function is applied to restore the corresponding original object identifier.

[0052] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0053] For a description of the features of the data traversal device in the embodiment of the storage system, please refer to the relevant description of the data traversal method in the embodiment of the storage system, which will not be repeated here.

[0054] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described data traversal method embodiments of the storage system.

[0055] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the data traversal method embodiments of any of the above storage systems when running.

[0056] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0057] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data traversal method embodiments of a storage system.

[0058] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the data traversal method embodiments of any of the above storage systems.

[0059] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0060] The foregoing has provided a detailed description of a data traversal method, apparatus, electronic device, and storage medium for a storage system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data traversal method for a storage system, characterized in that, include: Obtain the original object identifier, generate a corresponding converted identifier for each original object identifier through a conversion function and store it, wherein the conversion function is configured to make the set of converted identifiers have a more uniform distribution characteristic than the set of original object identifiers under a predetermined sorting rule. According to the sorting rules of the converted identifiers, the set of converted identifiers is divided into multiple consecutive and non-overlapping identifier intervals. For each of the identifier ranges, an independent traversal subtask is initiated. In each of the traversal subtasks, the list interface of the distributed object storage system is called to traverse the range based on the boundary information of the identifier range. The traversal results of all the traversal subtasks are collected, and for each transformed identifier in the traversal results, the inverse function of the transformation function is applied to restore the corresponding original object identifier.

2. The method according to claim 1, characterized in that, The transformation function is a hash function; The hash function is configured to: receive the byte sequence of the original object identifier as input; generate a corresponding hash value; select a character from a predetermined ordered character set as a prefix based on the hash value; and concatenate the prefix with the original object identifier to form the converted identifier.

3. The method according to claim 1, characterized in that, The multiple independent traversal subtasks are executed in parallel.

4. The method according to claim 1, characterized in that, After the original object identifier is obtained through restoration, the method further includes: The original object identifiers are sorted according to the predetermined sorting rules.

5. The method according to claim 1, characterized in that, The inverse function of the transformation function is configured to remove a fixed-length prefix portion from the transformed identifier.

6. The method according to claim 1, characterized in that, The boundary information of the identifier intervals is determined by writing the marker objects corresponding to the boundaries of each interval to the distributed object storage system.

7. The method according to claim 1, characterized in that, When a failure is detected in the traversal subtask, the failed traversal subtask is automatically restarted, and the traversal continues from the last successfully traversed identifier before the failure.

8. A data traversal device for a storage system, characterized in that, include: A generation unit is used to obtain the original object identifier, generate a corresponding converted identifier for each of the original object identifiers through a conversion function, and store it. The conversion function is configured to make the set of converted identifiers have a more uniform distribution characteristic than the set of original object identifiers under a predetermined sorting rule. A partitioning unit is used to divide the set of converted identifiers into multiple consecutive and non-overlapping identifier intervals according to the sorting rules of the converted identifiers. The traversal unit is used to start an independent traversal subtask for each of the identifier intervals. In each of the traversal subtasks, the list interface of the distributed object storage system is called to traverse the identifier interval based on the boundary information of the identifier interval. The restoration unit is used to collect the traversal results of all the traversal subtasks, and apply the inverse function of the transformation function to each transformed identifier in the traversal results to restore the corresponding original object identifier.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data traversal method of the storage system according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the data traversal method of the storage system according to any one of claims 1-7.

Images