A multi-cluster write scheduling method and system based on object identification coding

Abstract

The application discloses a multi-cluster write scheduling method and system based on object identification coding, the method comprising: allocating an atomic increment sequence identification to a storage bucket, determining a scheduling period for dividing a continuous scheduling period range according to a count threshold of the sequence identification, wherein the sequence identification is allocated by using a comparison and exchange operation at a hardware level to mark the write order of an object; routing the write request of the storage bucket to a fixed cluster within the scheduling period, determining a target cluster according to a dynamic count result of real-time resources of the cluster when the scheduling period terminates, the dynamic count result being used to represent the total amount of writeable resources of the cluster; updating a cluster identification segment in the object identification for representing the write cluster corresponding to the object and entering the next scheduling period. The application can directly complete routing analysis through the object identification, and optimizes the stability and transmission efficiency of multi-cluster write scheduling.

Description

Technical Field

[0001] This application belongs to the field of multi-cluster write scheduling technology, and in particular relates to a multi-cluster write scheduling method and system based on object identifier encoding. Background Technology

[0002] In existing distributed object storage multi-cluster write scheduling schemes, cluster routing for write requests often adopts real-time decision-making per request or static binding. Real-time decision-making per request leads to objects in the same bucket being stored across clusters, disrupting write affinity between buckets. Batch read operations require traversing data across clusters, reducing read efficiency. Static binding cannot adjust write targets based on real-time cluster resources, resulting in uneven cluster resource utilization and resource overload or idleness in some clusters. Furthermore, existing schemes often rely on distributed locks for sequence allocation, leading to lock contention and performance bottlenecks. They also require external metadata databases to store cluster ownership information, introducing single points of failure and consistency overhead. They cannot guarantee bucket write affinity while dynamically scheduling cluster resources, and it is difficult to achieve lock-free sequence allocation and stable scheduling without external metadata dependencies. Summary of the Invention

[0003] This application provides a multi-cluster write scheduling method and system based on object identifier encoding. During the multi-cluster write scheduling process, dynamic balanced scheduling is achieved based on the real-time resources of the cluster to ensure that the write objects of the same storage bucket are not scattered across the cluster to maintain write affinity. At the same time, lock-free sequence allocation and route resolution without external metadata dependency are achieved.

[0004] This application discloses a multi-cluster write scheduling method based on object identifier encoding, including: Atomically incrementing sequence identifiers are assigned to the storage bucket. The scheduling period for dividing a continuous range of scheduling periods is determined based on the counting threshold of the sequence identifiers. The sequence identifiers are assigned using hardware-level compare-and-swap operations and are used to mark the write order of objects. During the scheduling period, write requests from storage buckets are routed to fixed clusters. When the scheduling period ends, the target cluster is determined based on the dynamic counting result of the real-time resources of the cluster. This dynamic counting result is used to characterize the total amount of writable resources in the cluster. Update the cluster identifier segment in the object identifier that represents the corresponding object and enter the next scheduling cycle.

[0005] Optionally, assign atomically incrementing sequence identifiers to the bucket, including: Create a memory-level counter variable for the bucket; The value of the counter variable is updated by comparing and swapping operations; Use the value of the counter variable as the sequence identifier of the storage bucket; The counter variable is used to record the maximum value of the sequence identifiers allocated in the storage bucket. The memory-level counter variable is used to reduce the allocation latency of the sequence identifiers.

[0006] Optionally, the value of the counter variable can be updated through a comparison and swap operation, including: Get the current value of the counter variable; Increment the current value; The consistency of the count variable's value is verified by comparison and exchange operations. Once the value verification is passed, the incremented value is determined as the new count variable value. Numerical consistency is used to indicate that the count variable has not been modified by other threads.

[0007] Optionally, the scheduling period is determined based on a counting threshold of the sequence identifier, including: A preset, fixed counting threshold value; Match the sequence identifier with the counting threshold numerically; When the sequence identifier is within the counting threshold range, the current scheduling period is determined to be in a continuous state; when the sequence identifier reaches the counting threshold, the current scheduling period is determined to terminate. The counting threshold interval is used to divide a continuous range of scheduling cycles.

[0008] Optionally, the sequence identifier is numerically matched with a counting threshold, including: Extract the numerical content of the sequence identifier; Extract the numerical content of the counting threshold; The two numerical values ​​are compared, and the status of the scheduling cycle is determined based on the comparison result. Numerical content is used to characterize the quantization parameters of sequence identifiers and counting thresholds.

[0009] Optionally, when the scheduling period ends, the target cluster is determined based on the dynamic counting results of the cluster's real-time resources, including: Collect the remaining storage capacity, remaining processing performance, and operating status parameters of the cluster; Collect the remaining storage capacity, remaining processing performance, and running status parameters of the cluster to generate dynamic counting results; The dynamic counting results are sorted, and the cluster that ranks first in the sort is selected as the target cluster. Running status parameters are used to characterize whether the cluster is in a writable state.

[0010] Optionally, the remaining storage capacity, remaining processing performance, and running status parameters of the cluster can be collected to generate dynamic counting results, including: Assign quantization weights to remaining storage capacity, remaining processing performance, and operating status parameters; The parameters are numerically integrated based on their quantified weights; The integrated values ​​are used as the dynamic counting results; Quantized weights are used to differentiate resource reference priorities based on different parameters.

[0011] Optionally, update the cluster identifier segment in the object identifier and proceed to the next scheduling cycle, including: Fill the target cluster's identifier encoding into a fixed field of the object identifier; reset the start value of the sequence identifier's range; The write request for the next scheduling cycle is processed based on the updated cluster identifier segment; The fixed field is used to locate the storage location of the cluster identifier segment in the object identifier.

[0012] Optionally, write requests for the next scheduling cycle are processed based on the updated cluster identifier segment, including: Parse the object identifier corresponding to the write request; Extract the cluster identifier segment from the object identifier; The target cluster is located based on the cluster identifier segment, and the write request is routed to the target cluster to complete the data writing; Data writing is used to store the contents of an object to the storage medium of the target cluster.

[0013] This application also discloses a multi-cluster write scheduling system based on object identifier encoding, including: The sequence allocation module is used to allocate atomically incrementing sequence identifiers to storage buckets. The scheduling period used to divide the continuous scheduling period range is determined based on the counting threshold of the sequence identifiers. The sequence identifiers are allocated using hardware-level comparison and swap operations and are used to mark the writing order of objects. The period determination module is used to route write requests from storage buckets to fixed clusters within the scheduling period. When the scheduling period ends, the target cluster is determined based on the dynamic counting result of the real-time resources of the cluster. This dynamic counting result is used to characterize the total amount of writable resources of the cluster. The cluster scheduling module is used to update the cluster identifier segment in the object identifier that represents the object and to enter the next scheduling cycle.

[0014] As can be seen from the above technical solutions, this application avoids the use of distributed locks by assigning atomically incrementing sequence identifiers to storage buckets and relying on hardware-level comparison and exchange operations, thus eliminating the performance loss caused by lock contention. After dividing the scheduling period with the counting threshold of the sequence identifier, the fixed writing to the cluster within the scheduling period can concentrate objects in the same storage bucket, maintaining storage bucket write affinity. When the scheduling period ends, the target cluster is selected based on the dynamic counting result of the real-time resources of the cluster, which can adapt to the real-time changes of cluster resources and achieve balanced scheduling. At the same time, the cluster identifier is embedded in a fixed field of the object identifier, and the route resolution can be completed directly through the object identifier without the need for an external metadata database to store ownership information. Each technical feature works together in sequence during the execution process. The sequence identifier provides a quantitative basis for the period division, the period division ensures affinity, the dynamic counting achieves resource balancing, and the identifier embedding simplifies the routing process. The whole forms a scheduling system that is lock-free, has no external dependencies, and balances affinity and dynamic scheduling, thus solving the scheduling defects of the existing solutions. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a multi-cluster write scheduling method based on object identifier encoding in an embodiment of this application; Figure 2 This is a flowchart illustrating the process of allocating atomically incrementing sequence identifiers to storage buckets in a multi-cluster write scheduling method based on object identifier encoding, as described in an embodiment of this application. Figure 3 This is a flowchart illustrating the process of updating the value of a counter variable by comparing and exchanging operations in a multi-cluster write scheduling method based on object identifier encoding according to an embodiment of this application. Figure 4 This is a flowchart illustrating the process of determining the scheduling period based on the counting threshold of the sequence identifier in a multi-cluster write scheduling method based on object identifier encoding according to an embodiment of this application. Figure 5 This is a flowchart illustrating the process of matching sequence identifiers with counting thresholds in a multi-cluster write scheduling method based on object identifier encoding, as described in an embodiment of this application. Figure 6 This is a schematic diagram illustrating the process of determining the target cluster based on the dynamic counting results of real-time resources of the cluster when the scheduling cycle of a multi-cluster write scheduling method based on object identifier encoding terminates in an embodiment of this application. Figure 7This is a flowchart illustrating the process of collecting the remaining storage capacity, remaining processing performance, and running status parameters of a multi-cluster write scheduling method based on object identifier encoding in an embodiment of this application, and generating dynamic counting results. Figure 8 This is a schematic diagram of the process of updating the cluster identifier segment in the object identifier and entering the next scheduling cycle in a multi-cluster write scheduling method based on object identifier encoding according to an embodiment of this application. Figure 9 This is a flowchart illustrating the process of processing write requests for the next scheduling cycle based on the updated cluster identifier segment in a multi-cluster write scheduling method based on object identifier encoding, as described in an embodiment of this application. Figure 10 This is a schematic diagram of the structure of a multi-cluster write scheduling system based on object identifier encoding in an embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] In this application embodiment, distributed object storage systems are widely used in various data storage and management scenarios. Multi-cluster architecture can improve the system's storage capacity and processing capabilities. The write scheduling mechanism is a key link in the stable operation of multi-cluster systems. Existing multi-cluster write scheduling schemes have obvious defects in actual operation. Taking a distributed object storage system containing three storage clusters as an example, when the system receives continuous write requests from users for a single storage bucket, the existing scheduling scheme will calculate the target cluster for each write request in real time. The continuously written objects will be randomly assigned to different storage clusters. Objects in the same storage bucket will be in a cross-cluster distributed storage state. When users perform list read or batch read operations, the system needs to traverse multiple storage clusters to obtain data, which increases the time consumption of the read process and reduces the system's read efficiency. If a static binding method is used to associate the storage bucket with a single cluster, although write affinity can be maintained, it is impossible to adjust the write target in real time according to the remaining storage capacity, processing performance and other resources of the cluster. Some clusters will be overloaded due to continuous writing, while some clusters will be in an idle state, and the cluster resource utilization cannot reach a balanced state. Meanwhile, existing solutions rely heavily on distributed locks or central databases for sequence identifier allocation. When multiple threads write concurrently, lock contention occurs, increasing the latency of sequence allocation and limiting the system's concurrent processing capabilities. Furthermore, cluster ownership information needs to be stored in an external metadata database. If the metadata database fails, the routing function of the entire system will fail, posing a single point of failure risk. The synchronization and updating of metadata also introduces consistency overhead, increasing the system's operational complexity. These shortcomings mean that existing solutions cannot simultaneously meet the usage requirements of dynamic scheduling, write affinity, lock-free allocation, and no external dependencies.

[0019] This application addresses the shortcomings of existing technologies by proposing a multi-cluster write scheduling method and system based on object identifier encoding. The method optimizes scheduling through a process of sequence identifier allocation, scheduling cycle determination, dynamic cluster filtering, and object identifier updating. The following sections will elaborate on the technical features defined by the technical solution.

[0020] For example, this application discloses a multi-cluster write scheduling method, such as... Figure 1 As shown, the method includes: S100: Allocate atomically incrementing sequence identifiers to the storage bucket. Determine the scheduling period for dividing a continuous range of scheduling periods based on the counting threshold of the sequence identifiers. The sequence identifiers are allocated using hardware-level compare and swap operations and are used to mark the write order of objects.

[0021] S200: During the scheduling period, write requests from storage buckets are routed to fixed clusters. When the scheduling period ends, the target cluster is determined based on the dynamic counting result of the real-time resources of the cluster. This dynamic counting result is used to characterize the total amount of writable resources of the cluster.

[0022] S300: Update the cluster identifier segment in the object identifier that represents the corresponding write cluster and enter the next scheduling cycle.

[0023] This application provides a multi-cluster write scheduling method based on object identifier encoding. By assigning atomically incrementing sequence identifiers to storage buckets, relying on hardware-level comparison and exchange operations to avoid the use of distributed locks and eliminate the performance loss caused by lock contention, the scheduling period is divided by the counting threshold of the sequence identifier. Within the scheduling period, fixed write clusters can concentrate objects in the same storage bucket and maintain storage bucket write affinity. When the scheduling period ends, the target cluster is selected based on the dynamic counting results of the cluster's real-time resources. It can adapt to the real-time changes of cluster resources and achieve balanced scheduling. At the same time, the cluster identifier is embedded into a fixed field of the object identifier, and the route resolution can be completed directly through the object identifier without the need for an external metadata database to store ownership information.

[0024] Therefore, this application proposes a multi-cluster write scheduling method and system based on object identifier encoding. This method allocates atomically incrementing sequence identifiers to storage buckets through hardware comparison and exchange operations. Scheduling periods are divided based on the counting threshold of the sequence identifiers. During the scheduling period, the write cluster is locked to ensure storage bucket affinity. At the end of the period, real-time cluster resources are collected to generate dynamic counting results and filter target clusters. The target cluster identifier is then updated to a fixed field of the object identifier. In the next period, write requests are routed according to the updated identifier. Through lock-free allocation of sequence identifiers, periodic scheduling, and embedding cluster identifiers into object identifiers, dynamic balanced scheduling of cluster resources and storage bucket write affinity are achieved without using distributed locks or external metadata databases, thus improving the stability and efficiency of multi-cluster write scheduling.

[0025] The embodiments of this application will be described in detail below.

[0026] In this application, atomically incrementing sequence identifiers are the foundation for implementing periodic scheduling. Hardware-level compare-and-swap operations are the primary means of lock-free allocation. This operation relies on processor hardware instructions and does not require software-level distributed locks or central node coordination. When multiple threads concurrently request sequence identifiers, each thread can independently perform the compare-and-swap operation to obtain a unique sequence identifier value. The sequence identifier values ​​are continuous and non-repeating, corresponding to the write order of each object and providing a quantitative basis for the division of subsequent scheduling cycles. The counting threshold is a pre-set fixed value used to divide continuous sequence identifiers into multiple independent scheduling cycles. Within each scheduling cycle, the sequence identifier is within a fixed value range. At this time, all write requests from the bucket are routed to the same fixed cluster, which remains unchanged within the scheduling cycle. This process avoids the dispersion of objects from the same bucket across clusters, maintaining write affinity within the bucket. When the sequence identifier value reaches the counting threshold, the current scheduling cycle terminates. The system collects real-time resource parameters from all runnable clusters and converts these parameters into quantified dynamic counting results. These dynamic counting results directly reflect the total writable resources of each cluster. Based on these results, the system selects target clusters with sufficient resources, updates the target cluster's identifier to the cluster identifier segment of the object identifier, and then enters the next scheduling cycle. Write requests in the next cycle are all routed to the updated target cluster. The object identifier is composed of a cluster identifier segment and a sequence identifier segment. The cluster identifier segment is a fixed field in the object identifier, allowing direct parsing of target cluster information without the need for an external metadata database. This achieves routing resolution without external dependencies. The entire process involves no distributed lock contention and no external metadata overhead, while ensuring dynamic scheduling and write affinity.

[0027] In alternative implementations, such as Figure 2 As shown, the allocation of atomically incrementing sequence identifiers to the storage bucket includes: S110: Creates a memory-level counter variable for the bucket to record the maximum value of the allocated sequence identifiers within the bucket.

[0028] S120: Update the value of the counter variable by comparing and swapping operations.

[0029] S130: Use the value of the counter variable as the sequence identifier of the storage bucket.

[0030] In a specific example, each bucket independently creates a counter variable in the system's access node memory. This counter variable is a 64-bit long integer, sufficient to handle the sequence allocation of massive amounts of objects. The memory-level counter variable is stored in the access node's random access memory, where data read / write speeds are faster than disk storage or network transmission, reducing sequence identifier allocation latency and improving the system's concurrent processing capabilities. The initial value of the counter variable is set to zero. After each sequence identifier allocation, the counter variable is incremented. The counter variable continuously records the maximum value of allocated sequence identifiers within the bucket, ensuring that subsequently allocated sequence identifier values ​​are continuous and non-repeating. The comparison and swap operation is the main operation for sequence identifier allocation. This operation includes three parameters: the target variable in memory, the expected value, and the new value. During execution, the system compares the current value of the target variable with the expected value. If they match, the target variable is updated to the new value; otherwise, the update operation is abandoned. This operation is executed atomically at the hardware level and cannot be interrupted by other threads, ensuring value consistency in a multi-threaded concurrent environment without the need for locking. For example, when multiple threads simultaneously request the sequence identifier of the same storage bucket, each thread reads the current value of the counter variable as the expected value, increments the expected value by one to obtain the new value, and then performs a comparison and swap operation. Only one thread's expected value matches the current value of the counter variable. This thread successfully updates the value of the counter variable and obtains the sequence identifier. Other threads then reread the current value of the counter variable and perform the operation again until they successfully obtain the sequence identifier. This process involves no lock contention and no blocking wait.

[0031] In optional embodiments, such as Figure 3 As shown, updating the value of the counter variable through the comparison and swap operation includes: S210: Get the current value of the counter variable.

[0032] S220: Perform increment processing on the current value.

[0033] S230: Verify the consistency of the count variable's values ​​by comparing and exchanging operations. After the value verification is passed, the incremented value is determined as the new count variable value.

[0034] In a specific example, the system directly reads the value of the counter variable from memory. This reading process involves no network interaction or disk I / O, and the time is within microseconds. Incrementing the current value uses arithmetic addition, adding one to the current value to obtain the new value to be written. This operation is a fundamental processor instruction and is highly efficient. Value consistency verification is the key step in the comparison and exchange operation. The system compares the real-time value of the counter variable in memory with the current value obtained by the thread. If they are the same, it means the counter variable has not been modified by other threads during this operation, the value verification passes, and the system writes the incremented value to the counter variable, completing the value update. If they are different, it means the counter variable has been modified by other threads, the value verification fails, the current thread abandons the current operation, re-obtains the current value of the counter variable, and executes a new round of operations. This value verification mechanism ensures that in a multi-threaded concurrent environment, the sequence identifier value obtained by each thread is unique and continuous, preventing value duplication or gaps, and providing a stable numerical basis for the corresponding division of scheduling cycles.

[0035] In optional embodiments, such as Figure 4 As shown, determining the scheduling period based on the counting threshold of the sequence identifier includes: S310: Preset a fixed counting threshold.

[0036] S320: Match the sequence identifier with the counting threshold numerically.

[0037] S330: When the sequence identifier is within the counting threshold range, the current scheduling period is determined to be in a continuous state; when the sequence identifier reaches the counting threshold, the current scheduling period is determined to terminate.

[0038] In a specific example, the counting threshold is a fixed value preset during system initialization. This value can be configured based on the write frequency of the storage bucket and the resource size of the cluster. Storage buckets with higher write frequencies can be configured with larger counting thresholds, extending the duration of a single scheduling cycle and reducing the frequency of cluster switching. Storage buckets with lower write frequencies can be configured with smaller counting thresholds, shortening the duration of the scheduling cycle and improving scheduling flexibility. The counting threshold range is a continuous numerical range divided by the counting threshold. The range for the first scheduling cycle is from 1 to the counting threshold, the range for the second scheduling cycle is the counting threshold plus one to two times the counting threshold, and so on for subsequent cycles. The numerical range of each range is continuous and non-overlapping, completely covering the values ​​of all sequence identifiers. Matching the sequence identifier with the counting threshold means that the system obtains the value of the sequence identifier corresponding to the current write request in real time, determines the range in which the value falls. If the value is within the range of the current scheduling cycle, it means that the current scheduling cycle is ongoing, and the system continues to route the write request to the current fixed cluster. If the value equals the maximum value of the current range, that is, the counting threshold has been reached, it means that the current scheduling cycle terminates, and the system starts the cluster filtering process. This periodicity division mechanism relies on the numerical value of sequence identifiers, requiring no additional timing or triggering modules, resulting in a simple and highly stable process. The preset counting threshold does not depend on complex algorithms; it only needs to be configured according to the actual operating scenario of the system. For example, in an object storage system with three clusters, setting the counting threshold to 1000 allows the sequence identifier range for the first scheduling period to be 1 to 1000, the second period to 1001 to 2000, and so on. Each period can stably handle 1000 write requests, and the frequency of cluster switching matches the write traffic, preventing frequent switching or prolonged periods without switching. It should be noted that the specific values ​​of the counting threshold and the sequence identifier range for the scheduling period can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the inventive purpose of this application is achieved. This application does not impose any limitations on these settings.

[0039] In optional embodiments, such as Figure 5 As shown, the step of numerically matching the sequence identifier with the counting threshold includes: S410: Extract the numerical content of the sequence identifier.

[0040] S420: Extract the numerical content of the counting threshold.

[0041] S430: Compare the two numerical contents and determine the state of the scheduling cycle based on the comparison result.

[0042] In this specific example, the sequence identifier's value is an integer obtained through atomic increment, while the counting threshold's value is a preset fixed integer. Both values ​​are quantized parameters, allowing for direct size comparison. The comparison result has two scenarios: First, the sequence identifier's value is less than the counting threshold's value, in which case the sequence identifier is within the current scheduling period, and the scheduling period continues. Second, the sequence identifier's value equals the counting threshold's value, in which case the sequence identifier meets the period termination condition, and the scheduling period terminates. The system directly executes the corresponding scheduling operation based on the comparison result, without requiring complex logical judgments, resulting in high computational efficiency and adaptability to high-concurrency write scenarios.

[0043] In optional embodiments, such as Figure 6 As shown, when the scheduling cycle terminates, the target cluster is determined based on the dynamic counting results of the cluster's real-time resources, including: S510: Collect the remaining storage capacity, remaining processing performance, and operating status parameters of the cluster, and generate dynamic counting results. S520: Sort the dynamic counting results and select the cluster at the top of the sorted list as the target cluster.

[0044] In a specific example, after the scheduling cycle ends, the system needs to select clusters with sufficient resources as write targets for the next cycle. First, it collects real-time resource parameters for all clusters: remaining storage capacity (the amount of unused storage space in the cluster), remaining processing performance (the cluster's current write request handling capacity), and operational status (the cluster's working status). Only clusters in normal operating condition are selected as target clusters; clusters in rebalancing or fault states are excluded. This parameter collection process is implemented through the cluster monitoring interface, which collects real-time cluster operational data with low latency, reflecting the cluster's real-time resource status. Based on the collected parameters, a dynamic count result is generated. This dynamic count result is a quantified value, positively correlated with the total writable resources of the cluster; the larger the dynamic count result, the more abundant the total writable resources of the cluster. The system sorts the dynamic count results of all writable clusters in descending order and selects the cluster at the top of the sorted list as the target cluster. This cluster has sufficient resources and can handle write requests in the next cycle.

[0045] For example, the system comprises three normally operating clusters. Cluster A has a remaining storage capacity of 80TB and a remaining processing performance of 9000 IOPS; Cluster B has a remaining storage capacity of 50TB and a remaining processing performance of 5000 IOPS; and Cluster C has a remaining storage capacity of 70TB and a remaining processing performance of 8000 IOPS. After collecting these parameters, the system generates a dynamic counting result. Cluster A has the highest counting result and is ranked first, thus being identified as the target cluster for the next cycle. Write requests in the next scheduling cycle are all routed to Cluster A, making full use of the remaining resources of Cluster A and avoiding resource overload or idleness. It should be noted that the specific values ​​of storage capacity or processing performance in this application are merely illustrative. In practical applications, those skilled in the art can flexibly set them according to actual needs, as long as the purpose of this invention is achieved. This application does not limit this.

[0046] In optional embodiments, such as Figure 7 As shown, the remaining storage capacity, remaining processing performance, and operating status parameters of the acquisition cluster generate dynamic counting results, including: S610: Assign quantization weights to the remaining storage capacity, remaining processing performance, and operating status parameters. The quantization weights are used to distinguish the resource reference priorities of different parameters.

[0047] S620: Integrate the values ​​of each parameter according to the quantization weight.

[0048] S630: Use the integrated value as the dynamic counting result.

[0049] In this specific example, the quantization weights are fixed values ​​preset based on the importance of the parameters. Remaining storage capacity and remaining processing performance are the main parameters reflecting the writable resources of the cluster, and are assigned higher quantization weights. The running status parameters are filtering parameters, used only to determine whether the cluster is writable, and are assigned fixed quantization weights. The numerical integration process involves multiplying the value of each parameter by its corresponding quantization weight and summing the results. The sum is the dynamic counting result. This integration method highlights the influence of the main parameters, ensuring that the dynamic counting result accurately reflects the total writable resources of the cluster. The allocation of quantization weights does not require dynamic adjustment; it is configured during system initialization and remains stable during operation to avoid fluctuations in the dynamic counting result due to weight changes. The running status parameters are binary values: 1 for writable state and 0 for non-writable state. If the cluster is in a non-writable state, the numerical integration result is 0, and it will be automatically excluded from the filtering range, ensuring the availability of the target cluster.

[0050] In optional embodiments, such as Figure 8 As shown, updating the cluster identifier segment in the object identifier and entering the next scheduling cycle includes: S710: Fill the target cluster's identifier code into a fixed field of the object identifier.

[0051] S720: Reset the start value of the interval for the sequence identifier.

[0052] S730: Processes write requests for the next scheduling cycle based on the updated cluster identifier segment.

[0053] In this specific example, the object identifier uses a fixed-structure encoding format. The cluster identifier segment is located in the initial fixed field of the object identifier, and the sequence identifier segment follows the cluster identifier segment. The position and length of the fixed field are fixed, allowing direct location of the cluster identifier segment's content during parsing without traversing the entire object identifier. The target cluster's identifier is encoded as a unique integer, with each cluster corresponding to a fixed identifier code. After filling this code into the fixed field of the object identifier, the object identifier carries cluster affiliation information. Resetting the interval start value of the sequence identifier sets the interval start value of the next scheduling cycle to the current counting threshold plus one, ensuring the continuity of the cycle interval and matching the increasing value of the sequence identifier. When processing write requests based on the updated cluster identifier segment, the system does not need to query external metadata; it can directly parse the cluster identifier segment of the object identifier to locate the target cluster. The routing process is simple and has no external dependencies. The object identifier's encoding format is compatible with standard object storage protocols and will not affect normal client use. Clients can initiate write and read requests normally without modifying configurations or adapting processes. It should be noted that the encoding format of the object identifier does not require additional storage overhead. The length of the object identifier formed by combining the cluster identifier segment and the sequence identifier segment conforms to industry standards and will not increase the data transmission load. The parsing process relies on string truncation operations, which has high execution efficiency and will not affect the system's response speed.

[0054] In optional embodiments, such as Figure 9 As shown, processing the write request for the next scheduling cycle based on the updated cluster identifier segment includes: S810: Parse the object identifier corresponding to the write request.

[0055] S820: Extract the cluster identifier segment from the object identifier.

[0056] S830: Locate the target cluster based on the cluster identifier segment, and route the write request to the target cluster to complete the data write.

[0057] In a specific example, the write request includes information such as object identifier, object content, and bucket identifier. After receiving the write request, the system first parses the object identifier content, locates and extracts the cluster identifier segment through fixed fields, and the value of the cluster identifier segment corresponds to a unique target cluster. The system matches the corresponding cluster address based on this value to complete the target cluster location. The routing process is implemented through the system's internal network, transmitting the object content of the write request to the storage node of the target cluster. After receiving the object content, the target cluster stores it in the corresponding storage medium, completing the data write operation. The data write process relies on standard storage interfaces and is consistent with the write process of a single cluster, requiring no modification to the underlying storage logic of the cluster, thus ensuring strong compatibility. The processing flow of read requests is the same as that of write requests. After the client submits the object identifier, the system parses the cluster identifier segment and directly routes to the target cluster to read the data, without needing to traverse across clusters, maintaining the same read efficiency as in a single cluster scenario.

[0058] Specifically, the technical solution of this application is described in detail with reference to a practical operation case. In this case, the distributed object storage system includes three normally operating storage clusters: cluster 01, cluster 02, and cluster 03. Each cluster corresponds to a unique identifier code: cluster 01 is identified by code 01, cluster 02 by code 02, and cluster 03 by code 03. The system contains a business storage bucket with a preset counting threshold of 1000. The object identifier encoding format is a cluster identifier segment plus a sequence identifier segment, where the cluster identifier segment occupies 2 bits and the sequence identifier segment occupies 64 bits. During system initialization, a memory-level counting variable is created for this business storage bucket. The initial value of the counting variable is 0, the current scheduling cycle is the first cycle, the interval start value is 1, and the fixed cluster is cluster 01. When the system receives the first write request, it performs a comparison and swap operation to obtain the current value of the counter variable (0), increments it to 1, and updates the counter variable to 1 after successful value verification. The sequence identifier is set to 1, the object identifier to 011, and the write request is routed to cluster 01 to complete the data write. At this point, the sequence identifier 1 is within the range of 1 to 1000, and the first scheduling cycle continues. The system continuously receives write requests, and each request is assigned a consecutive sequence identifier through a comparison and swap operation. The cluster identifier segment of the object identifier remains 01. All objects are written to cluster 01 until the sequence identifier reaches 1000. When the sequence identifier reaches 1000, the value matching result indicates that the counting threshold has been reached, and the first scheduling cycle terminates. The system collects the remaining storage capacity, remaining processing performance, and running status parameters of clusters 01, 02, and 03, assigns quantized weights to each parameter, and generates a dynamic counting result. The dynamic counting result of cluster 02 is ranked first and is determined as the target cluster. The system fills the target cluster's identifier code 02 into the cluster identifier segment of the object identifier, resets the starting value of the sequence identifier range to 1001, and enters the second scheduling cycle. During the second scheduling cycle, the sequence identifier corresponding to the write request is between 1001 and 2000, and the cluster identifier segment of the object identifier is 02. All write requests are routed to cluster 02, maintaining write affinity for the storage bucket. When the sequence identifier reaches 2000, the second scheduling cycle terminates. The system then collects real-time cluster resources again, filters new target clusters, updates the cluster identifier segment, and enters the next cycle. When a client initiates a read request, it submits an object identifier such as 021500. The system parses the object identifier, extracts the cluster identifier segment 02, directly locates cluster 02, and reads the data without querying an external metadata database; the read process has no additional overhead. It should be noted that the specific values ​​of the technical threshold and identifier segment placeholders in this application are merely illustrative. In practical applications, those skilled in the art can flexibly set them according to actual needs, as long as the inventive purpose of this application is achieved. This application does not limit these settings.

[0059] Those skilled in the art will understand that, during operation, the atomically incrementing sequence identifier provides a corresponding quantitative basis for the scheduling cycle. Hardware-level comparison and exchange operations achieve lock-free sequence allocation, avoiding performance losses caused by lock contention. The scheduling cycle defined by the counting threshold can be fixedly written to the cluster, ensuring write affinity of the storage bucket and preventing objects from being stored scattered across the cluster. Dynamic counting and filtering at the end of the cycle can adapt to real-time cluster resources, achieving balanced utilization of cluster resources. The design of embedding the cluster identifier into the object identifier enables routing resolution without external metadata dependencies, eliminating the risk of single point of failure. The various technical features work together, requiring no distributed locks, no external metadata database, and no complex algorithm support, adapting to various distributed object storage multi-cluster scheduling scenarios.

[0060] Furthermore, the technical solution of this application is adaptable to cluster expansion and contraction scenarios. When a new storage cluster is added to the system, only a unique identifier code needs to be assigned to the new cluster. The system will automatically include the new cluster when collecting resource parameters, and write requests can be routed to the new cluster during dynamic counting and filtering, without modifying the underlying architecture of the system. When a cluster fails, the running status parameters will be marked as unwritable, and the system will automatically exclude the failed cluster and will not route write requests to the failed node, ensuring the reliability of data writing. A single storage bucket can support the writing of tens of billions of objects, meeting the needs of large-scale data storage scenarios. With memory-level counting variables and hardware-level atomic operations, the system's processing capacity can be linearly increased with the expansion of access nodes, without performance bottlenecks.

[0061] In summary, this application proposes a multi-cluster write scheduling method based on object identifier encoding. In the evolution of existing multi-cluster write scheduling technologies, maintaining write affinity between buckets and achieving dynamic resource balancing of clusters are generally considered two mutually constraining design goals. Real-time dynamic balancing scheduling requires recalculating the optimal cluster for each write request, which inevitably leads to objects from the same bucket being scattered across multiple clusters, failing to achieve centralized data locality. While static binding scheduling schemes can ensure that all objects within a bucket are written to the same cluster, they also lose the ability to respond to real-time changes in cluster resources, making it difficult to avoid situations where some clusters are overloaded while other clusters are idle.

[0062] This application introduces a scheduling cycle concept based on sequence counting, dividing a continuous stream of write requests into relatively independent processing units over time. Write affinity of a bucket does not necessarily require all historical objects to always belong to the same cluster; the key is that objects within a continuous write batch can be stored centrally. This application utilizes hardware-level compare-and-swap operations to assign atomically incrementing sequence identifiers to write requests in the bucket, providing a digital basis for lock-free sequence allocation. Subsequently, a preset counting threshold divides the incrementing sequence identifiers into continuous cycle intervals. Within a single cycle interval, all write requests are locked to a fixed cluster at the routing level, thus achieving centralized object storage and write affinity guarantees for the continuous write batches covered by that cycle. When the sequence identifier value exceeds the counting threshold of the current cycle, the system does not continue using the original cluster but triggers a cluster selection process. At this time, the remaining storage capacity and remaining processing performance of each cluster are collected and integrated into a dynamic counting result. Based on this result, the cluster with the most abundant current writable resources is selected as the target cluster for the next cycle. This process enables inter-cycle cluster switching and resource balancing scheduling.

[0063] At the implementation level, this application embeds the identifier encoding of the target cluster selected after the periodic switch directly into a fixed field of the object identifier, which, together with the sequence identifier segment, constitutes a complete object identifier. This combination of technical features allows the object identifier to carry complete routing and location information. For subsequent write or read requests for any object, the system only needs to parse the fixed field in its object identifier to directly locate the target cluster and complete data routing; the entire process does not require querying any external metadata database. This design, which internalizes the scheduling decision result into the data identifier, works closely with the periodic scheduling mechanism based on sequence counting, avoiding the contention overhead of distributed locks and eliminating the single point of dependency and consistency synchronization burden introduced by external metadata databases. From an overall technical perspective, this application, through the combination of the above technical features, achieves both write affinity guarantee and dynamic balancing of cluster resources without introducing additional heavyweight coordination components.

[0064] In this application embodiment, another aspect of the application discloses a multi-cluster write scheduling system based on object identifier encoding, such as... Figure 10 As shown, it includes: The sequence allocation module 11 is used to allocate atomically incrementing sequence identifiers to the storage bucket and determine the scheduling period for dividing the continuous scheduling period range based on the counting threshold of the sequence identifiers. The sequence identifiers are allocated by hardware-level comparison and swap operations and are used to mark the writing order of objects.

[0065] The period determination module 12 is used to route the write requests of the storage bucket to the fixed cluster within the scheduling period. When the scheduling period ends, the target cluster is determined based on the dynamic counting result of the real-time resources of the cluster. The dynamic counting result is used to characterize the total amount of writable resources of the cluster.

[0066] The cluster scheduling module 13 is used to update the cluster identifier segment in the object identifier that represents the object and write it to the cluster and enter the next scheduling cycle.

[0067] Based on the same principle, a specific implementation of the multi-cluster write scheduling system based on object identifier encoding in this application can be found in the implementation of the above method, and will not be described in detail here.

[0068] It should be noted that the technical solution in this application does not modify the underlying storage logic, protocol specifications, or client interfaces of existing object storage systems. It only optimizes the scheduling layer, ensuring seamless compatibility with existing S3 protocol-compatible distributed object storage systems. Users do not need to modify client configurations or redeploy storage clusters, reducing system modification and maintenance costs. The division of scheduling cycles, generation of dynamic counts, and parsing of object identifiers are all automated processes, requiring no manual intervention, reducing operational costs for maintenance personnel and enhancing the system's automated operation capabilities.

[0069] For example, compared with existing static binding schemes and real-time scheduling schemes, the technical solution of this application has significant advantages in write affinity, resource utilization, concurrency performance and availability. Static binding schemes cannot achieve dynamic resource scheduling, and real-time scheduling schemes destroy write affinity. However, the solution of this application solves the above problems by combining periodic scheduling and dynamic filtering. Furthermore, the lock-free allocation and metadata-free design further improve the stability and efficiency of the system, making it more suitable for large-scale, high-concurrency, multi-cluster object storage scenarios.

[0070] Specifically, in actual deployment, the technical solution of this application allows for adjustments to parameters such as counting thresholds and quantization weights according to business needs. These adjustments do not require a system restart and can be configured and effective online, enhancing system flexibility. Sequence identifier allocation, scheduling cycle determination, and cluster selection are all completed at the access node, eliminating the need for frequent interaction with the storage cluster. This reduces network transmission overhead, lowers the processing pressure on the storage cluster, and allows the storage cluster to focus on data storage and retrieval operations, thereby improving the overall system operating efficiency.

[0071] Those skilled in the art will understand that the technical solution of this application forms a complete multi-cluster write scheduling mechanism through lock-free atomic allocation of sequence identifiers, periodic division of counting thresholds, dynamic counting and filtering of cluster resources, and object embedding of cluster identifiers. Each technical feature is sequentially connected and cooperates with each other during operation. Sequence identifiers provide the basis for periodic division, periodic division ensures write affinity, dynamic filtering achieves resource balance, and identifier embedding simplifies the routing process. The overall solution is fully disclosed, and those skilled in the art can implement this technical solution based on the contents of the specification without creative effort. It can effectively solve the scheduling defects existing in the prior art and meet the actual usage needs of multi-cluster scheduling of distributed object storage.

[0072] In this embodiment, the sequence identifier allocation process is adaptable to distributed deployment scenarios with multiple access nodes. When the system has multiple access nodes, the counter variable of each storage bucket can be synchronized across multiple nodes through distributed shared memory. The synchronization process still uses a compare-and-swap operation, eliminating the need for distributed locks and ensuring the uniqueness and continuity of the sequence identifier in multi-node concurrent scenarios. The multi-access node deployment architecture can improve the system's concurrent processing capabilities while avoiding system unavailability caused by a single access node failure, thus improving the overall availability of the system.

[0073] It should be noted that the encoding format of the object identifier can be adjusted according to the actual deployment scenario, and the length of the cluster identifier segment can be configured according to the cluster size. When the number of clusters is small, the length of the cluster identifier segment can be shortened to reduce the overall length of the object identifier. When the cluster size is large, the length of the cluster identifier segment can be extended to ensure that each cluster corresponds to a unique identifier code. The adjustment process does not require modification of the system's scheduling logic; only the length parameter of the fixed field needs to be adjusted, making it highly adaptable.

[0074] For example, when the system's cluster size expands to 50 clusters, the length of the cluster identifier segment can be adjusted to a 2-digit decimal number, covering 100 clusters from 00 to 99, meeting the needs of 50 clusters. When the cluster size expands to 200 clusters, the length of the cluster identifier segment can be adjusted to a 3-digit decimal number, covering 1000 clusters from 000 to 999, meeting the needs of 200 clusters. After adjusting the length of the cluster identifier segment, the position of the fixed field is updated synchronously. During system parsing, the cluster identifier segment is extracted according to the updated fixed field length, without requiring modification to other process logic. It should be noted that the specific values ​​of the parameters mentioned above in this application are merely exemplary. In practical applications, those skilled in the art can flexibly set them according to actual needs, as long as the inventive purpose of this application is achieved. This application does not limit this.

[0075] Specifically, the technical solution of this application can be adapted to different types of storage business scenarios. For archive storage business, which has low write frequency and large data volume, a larger counting threshold can be configured to extend the scheduling cycle, reduce the frequency of cluster switching, ensure centralized storage of objects in the same archive dataset, and improve the efficiency of batch reading. For standard storage business, which has medium write frequency and balanced read and write operations, a medium counting threshold can be configured to balance scheduling flexibility and write affinity. For high-frequency access hotspot storage business, which has high write frequency and is sensitive to read latency, a larger counting threshold can be configured to enhance write affinity, ensure centralized storage of hotspot data, and reduce read latency.

[0076] Furthermore, the technical solution of this application is adaptable to cluster rebalancing scenarios. When a cluster enters a rebalancing state, its running status parameters are marked as unwritable. The system automatically excludes this cluster during dynamic counting and filtering, and will not select it as a target cluster. New write requests will not be routed to the rebalancing cluster, avoiding write traffic competing for cluster resources with the rebalancing process, ensuring the smooth execution of the rebalancing process. At the same time, front-end business writes are unaware and will not affect the normal operation of the business. When the cluster completes rebalancing, its running status parameters return to writable, and the system automatically includes it in the filtering range, deciding whether to select it as a target cluster based on the dynamic counting results. The entire process is automated and requires no manual intervention.

[0077] In this application embodiment, for multi-tenant use cases, the technical solution allows for independent configuration of parameters such as counting thresholds and quantization weights for each tenant's storage bucket. Storage buckets from different tenants are independent of each other, and scheduling processes do not interfere with each other, thus meeting the personalized business needs of different tenants. Tenant A's storage bucket can be configured with a large counting threshold to enhance write affinity. Tenant B's storage bucket can be configured with a small counting threshold to improve scheduling flexibility. The system can provide scheduling services to multiple tenants simultaneously without resource conflicts or performance interference.

[0078] It should be noted that the technical solution of this application, when processing read requests, does not need to traverse multiple clusters. It only needs to parse the cluster identifier segment in the object identifier to directly locate the target cluster. The time consumption of the read process is consistent with the single-cluster scenario, and the read latency will not be increased due to the multi-cluster architecture. For list read requests, each object identifier in the object list returned by the system carries a cluster identifier segment. Subsequent read requests for objects in the list can be directly parsed and routed without additional metadata queries. The efficiency of batch reads is consistent with the single-cluster scenario. In a specific example, when a user initiates a list read request for a storage bucket, the system returns a list of object identifiers, each of which carries a cluster identifier segment. When the user subsequently initiates batch read requests, the system can directly initiate read requests to the corresponding cluster in parallel based on the cluster identifier segment of each object identifier, without first querying metadata to determine object ownership. The batch read process is simple, efficient, and has low read latency. The technical solution of this application is compatible with the storage cluster's native multi-replica, erasure coding, and other data redundancy mechanisms. After the write request is routed to the target cluster, the target cluster stores the data according to the native redundancy mechanism without modifying the underlying data redundancy logic. The data reliability is consistent with the single-cluster scenario. Meanwhile, the multi-cluster architecture achieves fault domain isolation, so that the failure of a single cluster will not affect the normal operation of other clusters. The system can automatically route write requests to other normally operating clusters, ensuring business continuity and improving the overall reliability of the system.

[0079] Furthermore, the technical solution of this application enables seamless data migration in data migration scenarios. When it is necessary to migrate existing data in a storage bucket from cluster A to cluster B, the target cluster for the next scheduling cycle of that storage bucket can be directly set to cluster B. Newly written objects directly enter cluster B, and existing data can be migrated gradually in the background. During the migration process, there is no need to pause business writes or modify client configurations, and the business remains unaware of the migration. After the existing data migration is completed, all read requests are routed to cluster B. The entire migration process does not require modification of the system's scheduling logic, making the operation simple and the maintenance cost low.

[0080] Those skilled in the art will understand that the technical solution of this application decouples dynamic scheduling and write affinity in the time dimension through a periodic scheduling design, solving the problem that dynamic scheduling and write affinity cannot be simultaneously achieved in existing solutions. At the same time, it achieves lock-free sequence allocation through hardware-level atomic operations and achieves routing resolution without external metadata dependencies by embedding object identifiers with cluster identifiers. It solves the defects of existing solutions from multiple dimensions. The overall architecture is simple, highly stable, and highly adaptable, and can be widely applied to various distributed object storage multi-cluster scheduling scenarios.

[0081] In this embodiment, for high-concurrency write scenarios, the technical solution linearly improves the system's concurrent processing capacity by horizontally scaling access nodes. Each access node independently processes write requests and independently executes operations such as sequence identifier allocation, scheduling cycle determination, and cluster routing. Multiple access nodes synchronize counting variables through distributed shared memory. The synchronization process uses a compare-and-swap operation, eliminating lock contention and performance bottlenecks. When the number of concurrent write requests increases, only the number of access nodes needs to be increased to improve the system's processing capacity and meet the needs of high-concurrency business.

[0082] It should be noted that, in the operation of the technical solution of this application, all scheduling decisions are completed locally at the access node, without interaction with the central scheduling node. This avoids the performance bottleneck and single point of failure risk of the central scheduling node, resulting in higher overall system availability. Each access node can independently complete scheduling decisions, and even if some access nodes fail, other access nodes can still provide services normally without affecting the operation of the entire system.

[0083] For example, the system deploys 4 access nodes. One access node stops operating due to a hardware failure, while the remaining 3 access nodes can process write requests normally and independently complete operations such as sequence identifier allocation, scheduling cycle determination, and cluster routing. Business writes are uninterrupted, the system can operate normally, and maintenance personnel can repair or replace the faulty access node without affecting business operations.

[0084] Specifically, in terms of energy consumption optimization, the technical solution of this application can use dynamic counting and filtering to centrally route write requests to clusters with sufficient resources, avoiding multiple clusters operating at low load simultaneously. Idle clusters can be adjusted to low-power operation mode, reducing the overall energy consumption of the system and meeting the requirements of green and energy-saving use. When the load of a cluster increases, the system can automatically wake it up to normal operation mode, include it in the filtering range, and accept write requests. The entire process is completed automatically without manual intervention and does not affect the normal operation of business.

[0085] Furthermore, the technical solution of this application is adaptable to multi-cluster deployment scenarios across regions. Clusters in different regions can be included in the same scheduling system. When selecting target clusters, the system can incorporate parameters such as network latency and regional affiliation into the generation process of dynamic counting results, assign corresponding quantitative weights to parameters of different regions, and prioritize routing write requests to clusters in the user's region, reducing access latency and improving user experience. In multi-cluster deployment scenarios across regions, this solution can still guarantee write affinity within the same storage bucket. Write objects in the same region are centrally stored in the corresponding region's cluster, eliminating the need for cross-regional data transfer during reading, significantly reducing cross-regional network bandwidth consumption and improving read efficiency.

[0086] Those skilled in the art will understand that the technical solution of this application does not depend on specific hardware devices, specific operating systems or specific storage software, and can be deployed and run on various general-purpose server hardware, mainstream operating systems and standard distributed object storage software. It has a wide range of adaptability and strong practicality, and does not require large-scale hardware or software modifications, and can be quickly applied to actual production environments.

[0087] In this application embodiment, addressing the requirements for data compliance and data sovereignty, the technical solution can configure the scheduling cycle to route write requests from specific storage buckets to compliant regional clusters, ensuring the regional compliance of data storage. Simultaneously, by configuring a counting threshold, clusters that do not meet compliance requirements can be prevented from being selected as target clusters, ensuring that data is always stored within a compliant geographical area, thus satisfying the relevant requirements for data compliance and data sovereignty. Compliance configuration can be set independently for each storage bucket, and the compliance requirements of different storage buckets do not interfere with each other. When selecting target clusters, the system automatically excludes clusters that do not meet compliance requirements, ensuring that data storage always complies with compliance rules.

[0088] It should be noted that the technical solution of this application records operational data such as cluster switching information, sequence identifier allocation information, and write request routing information for each scheduling cycle during operation. This data can be used for system operation and maintenance monitoring, fault diagnosis, performance optimization, and other scenarios. Operation and maintenance personnel can view the system's scheduling status, cluster resource utilization, and write request processing status through the operational data, promptly identify problems in the system operation process, and make targeted optimizations and adjustments to improve the system's operational stability and performance.

[0089] For example, if operations personnel discover through operational data that a certain storage bucket is switching clusters too frequently, they can increase the counting threshold of that storage bucket, extend the scheduling cycle, reduce the frequency of cluster switching, and improve write affinity. If they find that the resource utilization of a certain cluster is consistently low, they can adjust the quantization weight to increase the priority of that cluster in the dynamic counting results, increase the write traffic that the cluster can handle, and improve resource utilization.

[0090] Specifically, in troubleshooting scenarios, the technical solution of this application can quickly locate the storage cluster of an object through the cluster identifier segment in the object identifier, without needing to query complex metadata. This allows for rapid troubleshooting of issues such as object read failures and data loss, improving the efficiency of troubleshooting. When a user reports that an object cannot be read, operations and maintenance personnel can directly locate the cluster where the object resides through the cluster identifier segment of the object identifier and troubleshoot that cluster without having to traverse all clusters, significantly shortening the troubleshooting time.

[0091] For example, in an edge computing scenario within a smart park, edge clusters are deployed within the park to store data such as surveillance videos and access control records, while a central cluster is deployed in the city's data center to store aggregated data for the entire park. The system can route real-time write requests from within the park to the edge clusters based on their resource availability, ensuring low latency for both writing and reading. Simultaneously, it ensures that video data from the same monitoring device is centrally stored on the edge clusters, maintaining write affinity and improving video playback efficiency. When the edge clusters' storage space is insufficient, the system can automatically route new write requests to the central cluster, ensuring normal business operation. Simultaneously, the system synchronizes existing data from the edge clusters to the central cluster, freeing up storage space on the edge clusters. The entire process is automated and requires no manual intervention.

[0092] Specifically, the technical solution of this application ensures that objects of the same training dataset are centrally stored in the same cluster in the scenario of artificial intelligence training data storage, thereby improving the efficiency of batch reading of training data and accelerating model training. In artificial intelligence training scenarios, frequent batch reading of training datasets is required. If the objects of the dataset are stored scattered across clusters, the latency of batch reading will increase significantly, affecting the efficiency of model training. This solution, through a periodic scheduling design, ensures that objects of the same dataset are centrally stored in the same cluster. Batch reading does not require cross-cluster traversal, significantly improving reading efficiency and effectively accelerating model training. Simultaneously, the system can dynamically adjust the target cluster for writing based on the resource status of the training cluster, ensuring that training data is stored in a cluster close to the training nodes, reducing network latency for data reading, and further improving training efficiency.

[0093] Furthermore, the technical solution of this application ensures that datasets for the same analysis task are centrally stored in the same cluster in big data analysis storage scenarios, improving data reading efficiency and shortening execution time. Big data analysis tasks require batch reading and processing of massive amounts of data, and data reading efficiency directly affects the execution time of the analysis task. The write affinity design of this solution ensures that datasets for the same analysis task are centrally stored, resulting in high batch reading efficiency and effectively shortening the execution time of the analysis task. Simultaneously, the system can dynamically schedule write requests based on the resource availability of the analysis cluster, ensuring that data is stored in the cluster where the analysis task resides, avoiding cross-cluster data transmission, reducing network bandwidth consumption, and improving the execution efficiency of the analysis task.

[0094] Those skilled in the art will understand that the technical solution of this application can be widely applied to various scenarios that require multi-cluster object storage, including cloud storage, big data analysis, artificial intelligence training, edge computing, video surveillance, data archiving, etc. It has strong adaptability and can effectively solve the problem of multi-cluster write scheduling in various scenarios, and has high practical value.

[0095] In this embodiment, the technical solution improves data security by encrypting the cluster identifier segment, preventing unauthorized access to the object's storage cluster information. The cluster identifier segment is encoded using an encryption algorithm, ensuring that only the system can parse the encrypted segment. External personnel cannot obtain the object's storage cluster information through the object identifier, preventing data storage location leakage and enhancing data security. The encrypted cluster identifier segment does not affect internal system parsing and routing; the system can correctly parse the encrypted segment, locate the target cluster, and complete write and read operations without modifying the scheduling process.

[0096] It should be noted that, in terms of data consistency, the technical solution of this application ensures that all objects within the same scheduling cycle are written to the same cluster through atomically incrementing sequence identifiers and periodic cluster switching. This prevents objects from being stored across different clusters, thus guaranteeing data consistency and integrity. Simultaneously, the cluster identifier segment of the object identifier is written synchronously with the object data, preventing inconsistencies between metadata and object data. This avoids the consistency overhead and risks associated with external metadata databases, resulting in higher data consistency.

[0097] For example, a user initiates write requests for 1000 objects in the same business batch. These 1000 objects are within the same scheduling cycle and are all written to the same cluster, avoiding cross-cluster storage. Subsequent batch read and processing operations for this batch of objects can be completed directly within the same cluster without cross-cluster operations, ensuring data consistency and processing efficiency. In system upgrade and version iteration scenarios, the technical solution of this application enables seamless rolling upgrades. Upgrades do not require pausing business writes or migrating data, and the business remains unaware of the changes. When the system's scheduling module needs to be upgraded, some access nodes can be upgraded first. Upgraded access nodes can process write requests normally and share counting variables and scheduling information with un-upgraded access nodes, avoiding duplicate sequence identifiers and cluster switching confusion. Once all access nodes are upgraded, the entire system version iteration is completed. The entire process requires no pausing of business writes, no modification to the storage cluster configuration, and the business remains unaware of the changes, ensuring system availability.

[0098] Furthermore, in disaster recovery scenarios, the technical solution of this application can achieve multi-cluster disaster recovery backup of data through a multi-cluster scheduling mechanism. When the primary cluster fails, the system can automatically route new write requests to the standby cluster to ensure business continuity. At the same time, the existing data in the primary cluster can be pre-synchronized to the standby cluster to achieve data disaster recovery backup. The entire disaster recovery switchover process is completed automatically without manual intervention, and the business is unaware of it, thus improving the system's disaster recovery capability. Meanwhile, the write affinity design of this solution ensures that objects for the same business are centrally stored in the primary and standby clusters. After the disaster recovery switchover, data reading and writing remain affinity-based, without affecting business performance.

[0099] Those skilled in the art will understand that the technical solution of this application, through a series of technical designs, solves several defects in existing multi-cluster write scheduling schemes, achieves a balance between dynamic balanced scheduling and write affinity, and realizes lock-free sequence allocation and route resolution without external metadata dependency, thereby improving the system's concurrency performance, availability, stability and ease of operation and maintenance. The overall solution has outstanding technical advantages and practical value.

[0100] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multi-cluster write scheduling method based on object identifier encoding, characterized in that, include: Atomically incrementing sequence identifiers are assigned to the storage bucket. The scheduling period for dividing a continuous range of scheduling periods is determined based on the counting threshold of the sequence identifiers. The sequence identifiers are assigned using hardware-level compare-and-swap operations and are used to mark the write order of objects. During the scheduling period, write requests from storage buckets are routed to fixed clusters. When the scheduling period ends, the target cluster is determined based on the dynamic counting result of the real-time resources of the cluster. This dynamic counting result is used to characterize the total amount of writable resources in the cluster. Update the cluster identifier segment in the object identifier that represents the corresponding object and enter the next scheduling cycle.

2. The method according to claim 1, characterized in that, Assign atomically incrementing sequence identifiers to the bucket, including: Create a memory-level counter variable for the bucket to record the maximum value of the allocated sequence identifiers within the bucket; The value of the counter variable is updated by comparing and swapping operations; Use the value of the count variable as the sequence identifier of the storage bucket.

3. The method according to claim 2, characterized in that, The value of the counter variable is updated through comparison and swap operations, including: Get the current value of the counter variable; Increment the current value; The consistency of the count variable's value is verified by comparison and exchange operations. Once the value verification is passed, the incremented value is determined as the new count variable value.

4. The method according to claim 1, characterized in that, The scheduling period is determined based on a counting threshold of the sequence identifier, including: A preset, fixed counting threshold value; Match the sequence identifier with the counting threshold numerically; When the sequence identifier is within the counting threshold range, the current scheduling period is determined to be in a continuous state; when the sequence identifier reaches the counting threshold, the current scheduling period is determined to terminate.

5. The method according to claim 4, characterized in that, The sequence identifier is numerically matched with the counting threshold, including: Extract the numerical content of the sequence identifier; Extract the numerical content of the counting threshold; The two numerical values ​​are compared, and the status of the scheduling cycle is determined based on the comparison result.

6. The method according to claim 1, characterized in that, When the scheduling cycle ends, the target cluster is determined based on the dynamic counting results of the cluster's real-time resources, including: Collect the remaining storage capacity, remaining processing performance, and running status parameters of the cluster to generate dynamic counting results; The dynamic counting results are sorted, and the cluster that ranks first in the sort is selected as the target cluster.

7. The method according to claim 6, characterized in that, The remaining storage capacity, remaining processing performance, and operating status parameters of the data acquisition cluster are used to generate dynamic counting results, including: Quantization weights are assigned to the remaining storage capacity, remaining processing performance, and operating status parameters. These quantization weights are used to distinguish the resource reference priorities of different parameters. The parameters are numerically integrated based on their quantified weights; The integrated values ​​are used as the dynamic counting result.

8. The method according to claim 1, characterized in that, Update the cluster identifier segment in the object identifier and proceed to the next scheduling cycle, including: Fill the target cluster's identifier encoding into a fixed field of the object identifier; Reset the start value of the interval for the sequence identifier; Write requests for the next scheduling cycle are processed based on the updated cluster identifier segment.

9. The method according to claim 8, characterized in that, Process write requests for the next scheduling cycle based on the updated cluster identifier segment, including: Parse the object identifier corresponding to the write request; Extract the cluster identifier segment from the object identifier; The target cluster is located based on the cluster identifier segment, and the write request is routed to the target cluster to complete the data write.

10. A multi-cluster write scheduling system based on object identifier encoding, characterized in that, include: The sequence allocation module is used to allocate atomically incrementing sequence identifiers to storage buckets. The scheduling period used to divide the continuous scheduling period range is determined based on the counting threshold of the sequence identifiers. The sequence identifiers are allocated using hardware-level comparison and swap operations and are used to mark the writing order of objects. The period determination module is used to route write requests from storage buckets to fixed clusters within the scheduling period. When the scheduling period ends, the target cluster is determined based on the dynamic counting result of the real-time resources of the cluster. This dynamic counting result is used to characterize the total amount of writable resources of the cluster. The cluster scheduling module is used to update the cluster identifier segment in the object identifier that represents the object and to enter the next scheduling cycle.

Images