Metadata cluster expansion method, device, equipment, medium and program product

Abstract

The application discloses a metadata cluster expansion method, device, equipment, medium and program product; the method comprises the following steps: obtaining a system state snapshot of a metadata cluster by performing periodic storage processing on metadata information of the metadata cluster; obtaining an operation log entry, the operation log entry comprising data change operation records sorted by time after the storage of the system state snapshot; determining whether the metadata cluster is expanded based on running data of the metadata cluster; if the metadata cluster is expanded, a first node is determined, the first node being a node to be added to the metadata cluster; and performing information synchronization on the first node based on the system state snapshot and the operation log entry, so that the first node after the information synchronization is consistent with the system state of the metadata cluster; thus, the expansion process is realized.

Description

Technical Field

[0001] This application relates to the field of object storage system technology in the cloud computing field, and in particular to a method, apparatus, device, computer-readable storage medium and computer program product for expanding a metadata cluster. Background Technology

[0002] Currently, the main method for scaling up object storage clusters is through manual observation. This manual approach has limitations: it cannot automatically detect scaling needs based on cluster status, frequent manual intervention increases labor costs, and decision-making delays can lead to unnecessary resource waste or service quality degradation.

[0003] In existing technologies, adding a new node to the cluster involves downloading the entire dataset from existing nodes through full synchronization. This process is time-consuming and consumes a significant amount of network bandwidth and storage resources. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a method, apparatus, device, computer-readable storage medium, and computer program product for expanding a metadata cluster.

[0005] The method for expanding the metadata cluster provided in this application includes:

[0006] The metadata information of the metadata cluster is periodically stored to obtain a system state snapshot of the metadata cluster;

[0007] Obtain operation log entries, which include data change operation records sorted by time after storing the system state snapshot;

[0008] Based on the operational data of the metadata cluster, it is determined whether the metadata cluster needs to be expanded; if it needs to be expanded, a first node is identified, which is the node to be added to the metadata cluster.

[0009] The first node is synchronized based on the system status snapshot and the operation log entries so that the synchronized first node is consistent with the system status of the metadata cluster.

[0010] The metadata cluster expansion method provided in this application embodiment, applied to the first node, includes:

[0011] Obtain a system status snapshot, the system status snapshot including a timestamp and metadata information corresponding to the timestamp;

[0012] Store the system state snapshot and construct the initial system state of the first node based on the system state snapshot;

[0013] Synchronize operation log entries from a second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster;

[0014] The operation log entries are applied one by one to update the initial system state until the first node is consistent with the system state of the metadata cluster.

[0015] The metadata cluster expansion device provided in this application embodiment is applied to a metadata cluster, and the device includes:

[0016] A data storage unit is used to periodically store the metadata information of the metadata cluster to obtain a system state snapshot of the metadata cluster.

[0017] The data acquisition unit is used to acquire operation log entries, which include data change operation records sorted by time after storing the system state snapshot;

[0018] The data processing unit is used to determine whether the metadata cluster needs to be expanded based on the running data of the metadata cluster; if expansion is required, a first node is determined, which is a node to be added to the metadata cluster; the first node is used to synchronize information based on the system status snapshot and the operation log entries, so that the synchronized first node is consistent with the system status of the metadata cluster.

[0019] The metadata cluster expansion device provided in this application embodiment is applied to a first node, and the device includes:

[0020] A data acquisition unit is used to acquire a system status snapshot, the system status snapshot including a timestamp and metadata information corresponding to the timestamp;

[0021] A data storage unit is used to store the system state snapshot and construct the initial system state of the first node based on the system state snapshot;

[0022] A data synchronization unit is used to synchronize operation log entries from a second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster; the operation log entries are applied one by one to update the initial system state until the system state of the first node is consistent with that of the metadata cluster.

[0023] The electronic device provided in this application includes a processor and a memory. The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to execute any of the above-described methods for expanding the metadata cluster.

[0024] The computer-readable storage medium provided in this application embodiment is used to store a computer program that causes a computer to execute any of the above-described methods for expanding the metadata cluster.

[0025] The computer program product provided in this application includes computer program instructions that cause a computer to execute any of the above-described methods for expanding the metadata cluster.

[0026] In the technical solution of this application embodiment, a system state snapshot of the metadata cluster is obtained by periodically storing the metadata information of the metadata cluster; operation log entries are obtained, which include data change operation records sorted by time after storing the system state snapshot; it is determined whether the metadata cluster needs to be expanded based on the running data of the metadata cluster; if expansion is required, a first node is determined, which is a node to be added to the metadata cluster; information synchronization of the first node is performed based on the system state snapshot and operation log entries, so that the synchronized first node is consistent with the system state of the metadata cluster; in this way, it is possible to automatically determine whether expansion is needed based on the current cluster running data, and to realize the automatic expansion process. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the structure of an object storage system provided in an embodiment of this application;

[0028] Figure 2 This is a schematic diagram illustrating the principle of a metadata cluster expansion method provided in an embodiment of this application;

[0029] Figure 3 This is a flowchart illustrating a method for scaling up a metadata cluster provided in an embodiment of this application. Figure 1 ;

[0030] Figure 4 This is a flowchart illustrating a method for scaling up a metadata cluster provided in an embodiment of this application. Figure 2 ;

[0031] Figure 5 This is a flowchart illustrating a method for scaling up a metadata cluster provided in an embodiment of this application. Figure 3 ;

[0032] Figure 6 This is a flowchart illustrating a method for scaling up a metadata cluster provided in an embodiment of this application. Figure 4 ;

[0033] Figure 7 This is a flowchart illustrating a method for scaling up a metadata cluster provided in an embodiment of this application. Figure 5 ;

[0034] Figure 8 This is a schematic diagram of the structural composition of the metadata cluster expansion device provided in the embodiments of this application. Figure 1 ;

[0035] Figure 9 This is a schematic diagram of the structural composition of the metadata cluster expansion device provided in the embodiments of this application. Figure 2 ;

[0036] Figure 10 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0041] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0042] 1) Object-based storage is a massive, object / file-oriented internet storage system, also known as "cloud storage." An object storage system is a data storage architecture that packages data, metadata, and a unique identifier into a unit called an "object," and stores it in a flat, infinitely scalable storage pool. Object storage systems employ a scale-out architecture, allowing for the storage of massive amounts of unstructured data by continuously adding storage nodes.

[0043] 2) The metadata cluster, also known as the object storage index cluster, is the "brain" or "directory center" of the object storage system. It is not a cluster that stores actual file data (such as pictures and videos), but a highly available and scalable distributed system that is specifically responsible for managing the metadata and global namespace of all files.

[0044] 3) An object is the basic unit of data storage in an object storage system. An object is actually a combination of file data and a set of attribute information. All objects have an object identifier, and the object is accessed through the object identifier command.

[0045] In related technologies, the mainstream method for scaling up object storage cluster services relies on manual observation and maintenance, failing to achieve automatic detection and scaling based on cluster status. Frequent manual intervention increases labor costs and may also lead to unnecessary resource waste or service quality degradation due to decision-making delays. Furthermore, downloading the entire dataset from existing nodes during full synchronization when nodes join the cluster is not only time-consuming but also consumes significant network bandwidth and storage resources, impacting system scalability and response speed. Traditional architectures often create strong dependencies between the gateway and backend index nodes, requiring manual updates to configuration files and service restarts for each change to take effect, severely limiting system flexibility and scalability.

[0046] To address the aforementioned issues, embodiments of this application provide a method, apparatus, device, computer-readable storage medium, and computer program product for expanding a metadata cluster, which can improve the system's flexibility and scalability.

[0047] Taking the application of this application's embodiments to object storage services as an example, see [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the structure of an object storage system provided in an embodiment of this application. Object storage is a new network storage architecture, exemplified by... Figure 1The system involves an object storage service 100 and a client 200. The client 200 connects to the object storage service 100 via a network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. An object is the basic unit of data storage in the system. An object is essentially a combination of file data and a set of attribute information (meta data). This attribute information can define file-based parameters, data distribution, and quality of service, etc. In a storage device, every object has an object identifier, which is accessed via OSD commands. There are typically several types of objects: the root object on the storage device identifies the storage device and its various attributes; group objects are collections of objects on the storage device that share resource management policies, etc.

[0048] Here, the object storage service 100 includes an object storage cluster 101. The object storage cluster 101 includes an object-based storage device 1011 (OSD), which has its own central processing unit (CPU), memory, network, and disk system. The main functions of the OSD include data storage and secure access. The OSD provides three main functions: 1) Data storage: The OSD manages object data and places it on a standard disk system. The OSD does not provide a block interface access method; clients request data using object IDs and offsets for data read and write. 2) Intelligent distribution: The OSD optimizes data distribution using its own CPU and memory and supports data prefetching. Because the OSD can intelligently support object prefetching, it can optimize disk performance. 3) Management of metadata for each object: The OSD manages the metadata of the objects stored on it, typically including the object's data blocks and object length. The object storage architecture delegates the main metadata management work in the system to the OSD, reducing client overhead.

[0049] Here, the object storage service 100 includes a metadata cluster 102. The metadata cluster 102 can be a metadata server (MDS) or a server cluster. The metadata cluster 102 can also be called an object storage index service. As the metadata cluster of the object storage system, it carries the index information of all objects in the cluster. Objects uploaded by the client 200 must be accessed directly through a globally unique identifier stored in the cluster. This globally unique identifier allows for quick location and retrieval of these objects without relying on traditional file paths or block addresses. This enables the object storage system to efficiently find and return the corresponding data object based on the object ID. The metadata cluster 102 connects multiple servers 1021 (nodes) to work collaboratively, forming a unified storage resource pool. To ensure high availability of the index service, the index data of the same object is replicated into multiple copies and distributed across different nodes in the cluster. Each node in the cluster participates in storage and provides data services.

[0050] In some embodiments, the server is a server cluster or distributed system consisting of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The cloud services can be interactive processing services that can be invoked by the terminal.

[0051] See Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of the metadata cluster expansion method provided in this application embodiment. The method consists of a monitoring and computing layer, a control layer, an index service layer, and a load balancing gateway layer. The monitoring and computing layer obtains the current operating data of the metadata cluster, and the control layer issues expansion commands to automatically implement the expansion process. By using an elastic load balancer (SLB) to connect the gateway and the index service, network link configuration decoupling is achieved, enabling hot updates of index node addresses. During the expansion process, the observation panel component calls the index cluster status interface to summarize various cluster information, achieving complete visualization of the cluster's operating status and the expansion process.

[0052] See Figure 3 , Figure 3 This is a flowchart illustrating the capacity expansion method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0053] Step 301: Perform periodic storage processing on the metadata information of the metadata cluster to obtain a snapshot of the system status of the metadata cluster.

[0054] In this embodiment, the metadata cluster is a server cluster in the object storage system used for object storage indexing services, carrying the index information of all objects in the cluster. Objects uploaded by clients must be accessed directly through a globally unique identifier stored in the metadata cluster, allowing for quick location and retrieval of these objects. The metadata cluster connects multiple nodes together to work collaboratively, forming a unified storage resource pool.

[0055] In some embodiments, high availability of the index service is ensured by replicating the index data of the same object into multiple copies and distributing them across different nodes in a metadata cluster, where each node in the metadata cluster participates in storing and providing data services.

[0056] In some embodiments, the metadata cluster includes a second node, which is an existing node in the metadata cluster. (See reference...) Figure 4 , Figure 3 Step 301 shown can be implemented through steps 401 to 402, which are explained in detail below.

[0057] Step 401: Obtain a system status snapshot generated by each second node based on a preset period. The system status snapshot includes a timestamp and the metadata information corresponding to the timestamp.

[0058] Here, the second node, also known as the index service node, is a dedicated server responsible for managing and storing metadata. Specifically, the second node stores the unique identifier, size, creation time, permissions, checksum, and physical storage location of each object within the cluster. When a user requests a file, the request first reaches the index service node. The node quickly returns the file's exact location by querying its internal metadata database, guiding the client or gateway to the correct location to retrieve the data.

[0059] Step 402: Store a system state snapshot of each second node to obtain a system state snapshot of the metadata cluster.

[0060] Here, the second node in the metadata cluster, i.e., the existing node, periodically (e.g., hourly or daily) generates a snapshot of the system state. This snapshot contains all metadata information at a specific point in time, similar to a full backup in a database. The snapshot is saved to the distributed file system in the object storage system or other persistent storage for use when the first node joins the metadata cluster.

[0061] For example, in an object storage metadata cluster, a second node, acting as an existing node, will automatically trigger the generation process of a system state snapshot, following a pre-configured unified period (e.g., hourly, daily, with the specific period dynamically adjusted based on business data volume and update frequency). Subsequently, the second node will traverse all metadata information stored within its own system, including but not limited to globally unique identifiers for each object, object attributes (such as size, creation time, and storage location mapping), and index relationships. This information will be integrated into a complete, time-consistent dataset. Simultaneously, the system will generate a unique timestamp for this snapshot. This timestamp not only marks the snapshot's generation time but is also bound to all metadata information within the snapshot, used to distinguish the time order of different snapshots later.

[0062] In some embodiments, the metadata cluster further includes a third node, which is determined by the metadata cluster based on the first protocol, and is used to manage the information synchronization between the first node and the second node.

[0063] Here, the third node is the leader node in the metadata cluster, also known as the seed node. The first protocol includes the Replicated And Fault Tolerant (Raft) protocol, which ensures data consistency and high system availability. The third node only commits an update after a majority of nodes have acknowledged receiving and persisting it, thus preventing data loss and inconsistency. If the current third node fails, the Raft protocol automatically triggers a new election to select a new leader to take over log replication and information synchronization management tasks. This incremental synchronization mechanism not only improves the initialization efficiency of new nodes but also enhances the fault tolerance and scalability of the entire system. In this way, the object storage system can smoothly scale without affecting existing services while maintaining data consistency and reliability.

[0064] For example, once the second node completes snapshot generation, it proactively sends a snapshot-ready notification to the system. This notification includes the second node's node identifier, snapshot timestamp, snapshot data digest information (such as a hash value for subsequent data integrity verification), and a temporary access address for the snapshot data. Upon receiving the notification, the system first performs a preliminary check on the validity of the snapshot's timestamp and the integrity of the data digest. If everything is correct, it sends a snapshot retrieval command to the second node. If the check fails (e.g., the timestamp deviates significantly from the current cluster time, or the digest does not match), the second node is triggered to regenerate the snapshot. Finally, the system uses the temporary access address provided by the second node to obtain a snapshot of the node's system state, while simultaneously recording the corresponding node identifier, timestamp, and other associated information to ensure that each second node's snapshot can be accurately identified and tracked.

[0065] In some embodiments, after obtaining system state snapshots of all second nodes, the cluster initiates a system state snapshot storage process to form a system state snapshot covering the entire metadata cluster.

[0066] Here, the system performs consistency alignment on all acquired second-node snapshots. Specifically, due to potential differences in hardware performance and load across different second-nodes, the actual time of snapshot generation for each node may have slight deviations. In this case, the system uses the earliest timestamp among all system state snapshots as the baseline timestamp for the cluster snapshot, ensuring that the final stored cluster snapshot reflects the metadata status of all second-nodes at the same point in time. If a second-node's snapshot timestamp is later than the baseline timestamp, the system further verifies whether there have been metadata changes for that node between the baseline timestamp and its own snapshot timestamp. If so, it rolls back to the state corresponding to the baseline timestamp based on the node's operation log, ensuring time consistency across all node snapshots. Next, the system stores the processed second-node snapshots in a pre-defined persistent storage system, typically a distributed file system such as the Hadoop Distributed File System (HDFS) or a high-performance object storage service. The storage process employs a sharding and replication strategy. Specifically, each second-node snapshot is split into several shards (e.g., 1GB per shard) based on data size, facilitating parallel downloading during subsequent new node synchronization and improving synchronization efficiency. Generate at least 2-3 replicas for each snapshot shard and distribute them across different physical nodes in the distributed storage system to avoid snapshot loss due to the failure of a single storage node and ensure high availability of snapshots.

[0067] In some embodiments, during the storage process, a unique storage identifier is generated for each system state snapshot shard, and a cluster snapshot index table is established. This cluster snapshot index table records key information such as the base timestamp of the cluster snapshot, all included second-node identifiers, shard information (shard ID, storage address, replica location) for each node's snapshot, and data digests. Simultaneously, the system performs integrity verification on the stored snapshot shards. By comparing the actual hash value of the shard with the previously recorded digest information, it confirms that the shard has not been corrupted during storage. If the verification fails, the shard is retrieved again from the corresponding second-node and stored again.

[0068] In some embodiments, once the system state snapshot shards of all second nodes have been stored and verified, the cluster snapshot index table can be marked as available, thus forming a complete metadata cluster system state snapshot. This cluster snapshot not only contains metadata information of all second nodes at the base timestamp, but also achieves unified management of snapshots for each node through the index table. When a new node joins the cluster, it can directly obtain the index information of the cluster snapshot through the third node, quickly locate and download the required snapshot shards, laying the foundation for the incremental synchronization mechanism.

[0069] In some embodiments, when the metadata cluster determines that it needs to be expanded, i.e., by adding a first node to the metadata cluster, the first node first requests the latest system state snapshot from the third node in the cluster. The third node returns the most recent system state snapshot and its timestamp. The first node then loads the returned system state snapshot into memory as the initial data state.

[0070] In some embodiments, before the second node generates a system state snapshot, the second node may temporarily freeze all data change operations (such as adding, deleting, and updating object indexes) within the current node to ensure that no new metadata changes interfere during snapshot generation and to guarantee the consistency of the snapshot data.

[0071] Step 302: Obtain operation log entries. Operation log entries are data change operation records sorted by time after the storage system status snapshot.

[0072] In some embodiments, all operations on the indexed data (such as adding, deleting, updating, etc.) since the last snapshot is generated are logged as operation log entries. These log entries are arranged in chronological order and contain sufficient information to reproduce the operations. Each operation log entry is immediately written to persistent storage media after being confirmed to have been executed, ensuring recovery even in the event of a failure.

[0073] In some embodiments, the system sends a log retrieval instruction to each second node in the metadata cluster. After receiving the log retrieval instruction sent by the system, the second node filters out all log entries whose operation timestamps are later than the cluster snapshot base timestamp from the local persistent log storage medium.

[0074] For example, after the metadata cluster completes the storage of the metadata cluster system state snapshot and marks the snapshot as available, the process of obtaining operation log entries will be triggered. At this time, the log collection scope is all second nodes (existing nodes) in the cluster. Each second node continues to process data change operations such as adding, deleting, and updating object indexes after the snapshot is generated, and each type of operation will generate a corresponding log entry. It is necessary to ensure that no node logs are missed. After receiving the log acquisition instruction, the second node will first filter all log entries whose operation timestamps are later than the cluster snapshot base timestamp from the local persistent log storage medium (such as the log partition built into the solid state drive (SSD) or the distributed log subsystem bound to the node). This filtering logic can accurately exclude historical operations before the snapshot is generated and only retain incremental changes that need to be synchronized after the snapshot.

[0075] The information included in the operation log entries includes, but is not limited to: operation type identifier, such as "ADD" for index addition, "DEL" for index deletion, and "UPD" for index attribute update; a globally unique identifier for the target object to ensure that the specific index can be located later; change details, such as complete metadata of the new index for add operations, a comparison of metadata before and after the change for update operations, and the association information of the index to be deleted for delete operations; operation execution timestamp for subsequent time sorting; operation execution status code, such as "200" for successful execution and "500" for failed execution, only successful logs will be extracted to avoid invalid change synchronization; and the second node ID that generated the log for easy tracing of the log source later.

[0076] In some embodiments, after filtering out the target log entries, each second node can perform format validation on individual log entries to ensure that no fields are missing and the timestamp format is compliant. Then, it calculates the overall hash value for all log entries to be submitted, and packages the set of operation log entries, the overall hash value, and the second node ID into a single file and sends it to the third node.

[0077] In some embodiments, after the system has completed the receipt and verification of logs from all second nodes, it will aggregate all qualified log entries into the global log integration pool and start the time sorting process.

[0078] For example, all logs are sorted in ascending order using the operation timestamp as the primary sorting dimension to ensure that subsequent application logs follow the actual operation sequence and avoid state confusion. If two logs have completely identical operation timestamps, such as in extreme scenarios where different nodes process different objects simultaneously, the ID of the second node that generated the log is used as the secondary sorting dimension to ensure that the sorting result is unique and reproducible. After sorting, the system adds a log batch identifier to the integrated log set, including the start timestamp, end timestamp, and total number of logs in the batch, and stores it in a cluster-level persistent log repository, associating it with the snapshot storage location. This facilitates one-stop retrieval of snapshots and logs when new nodes synchronize, and updates the cluster state to log ready, preparing for subsequent new nodes to catch up on the data state.

[0079] Step 303: Determine whether the metadata cluster needs to be expanded based on the running data of the metadata cluster; if it needs to be expanded, determine the first node, which is the node to be added to the metadata cluster.

[0080] In some embodiments, refer to Figure 5 , Figure 3 Step 303 shown can be implemented through the following steps 501 to 502, which are explained in detail below.

[0081] Step 501: Monitor the running data of the metadata cluster through the computing layer, and determine whether the running data is greater than the first threshold. If it is greater, determine that the metadata cluster needs to be expanded and send an expansion request to the control layer.

[0082] Here, the system also includes a computation layer, which can also be called a visualization layer or a monitoring computation layer. The computation layer is used to aggregate metadata cluster statistics into observable modules, including traffic statistics, transactions per second (TPS) statistics, and various node load information. After collecting and aggregating various system operating data of the index cluster, this layer calculates whether the current system's load capacity exceeds the cluster's design specifications. If it has exceeded the design limits, it sends a scaling request to the control layer.

[0083] For example, the computing layer collects multi-dimensional operational data in real time from all second-level nodes (existing nodes) and related components of the metadata cluster using a preset collection protocol. This multi-dimensional operational data includes business data, resource data, and cluster monitoring data. Specifically, business data includes, but is not limited to, inbound / outbound traffic for index queries (unit: Mbps), transactions per second (TPS, breaking down the success and failure rates of index additions / deletions / updates), and request response latency (statistical P90 / P95 / P99 quantile values, accurately reflecting user experience); resource data includes, but is not limited to, CPU utilization (%), memory usage (including metadata cache and service process usage), disk I / O read / write throughput (for metadata storage partitions), and network connection count (number of connections between currently active clients and nodes) for each second-level node; cluster monitoring data includes, but is not limited to, snapshot generation success rate, operation log synchronization latency (comparing the time difference between logs from different nodes), and error log generation frequency, such as the number of log entries per minute for index query failures and metadata write anomalies.

[0084] In some embodiments, the collected data can be categorized and stored in the database according to indicator type, node ID, and collection timestamp, and synchronized to the visualization dashboard in real time to form an observable operating status module.

[0085] In some embodiments, the first threshold is a pressure safety boundary preset based on the metadata cluster design specifications, which can be configured in conjunction with the Service Level Agreement (SLA) and the hardware capacity limit.

[0086] For example, when the TPS design peak is 6000 times / second, the first threshold is set to 5400 times / second (with 10% buffer space reserved); when the CPU utilization safety limit is 85%, the first threshold is set to 80%; when the response latency SLA requirement P95≤150ms, the first threshold is set to 130ms.

[0087] In some embodiments, the computing layer analyzes historical data every 24 hours. If a certain indicator remains below 60% of the threshold for an extended period (e.g., TPS consistently below 3000 times / second), the threshold is automatically lowered to avoid resource waste. If an indicator frequently approaches the threshold (e.g., CPU usage repeatedly reaches 78%-79%), manual review is triggered to confirm whether the threshold needs adjustment or capacity expansion is necessary. The verification process employs a multi-cycle continuous judgment logic. For core indicators (TPS, CPU utilization, response latency), if they exceed the first threshold for three consecutive collection cycles (30 seconds), and for non-core indicators (disk I / O, connection count), if they exceed the first threshold for five consecutive collection cycles (50 seconds), the cluster pressure is preliminarily determined to be excessive.

[0088] In some embodiments, after ruling out temporary anomalies, the computing layer determines that expansion is needed and generates an expansion request.

[0089] Here, the expansion request message includes a universally unique identifier (UUID), details of the over-limit metrics (e.g., "TPS = 5800 times / second > threshold 5400 times / second, CPU utilization = 82% > threshold 80%)", a snapshot of the current cluster status (number of nodes, resource utilization of each node), the suggested expansion size (calculated based on the over-limit extent, e.g., suggesting adding 2 more nodes), and a request timestamp. It is sent to the control layer via the cluster's internal Remote Procedure Call (RPC) protocol, while simultaneously initiating a 30-second timeout retry mechanism (e.g., retrying up to 3 times) to ensure that the expansion command is not lost.

[0090] Step 502: Receive the scaling request sent by the computing layer through the control layer, obtain configuration information from the metadata cluster, and perform instantiation processing based on the configuration information through container orchestration strategy to determine the first node.

[0091] Here, the system also includes a control layer. As the manager of the entire object storage cluster service, the control layer, while not directly involved in the storage process, plays a crucial role in the object storage cluster. The control layer controls the replicas, status, and configuration of the containers running on each component in the cluster by detecting and responding to cluster events. When the monitoring and compute layer issues a scaling command to the control layer, the control layer obtains information such as node tags, configurations, and image versions from the current cluster and transmits this configuration information to the node to be scaled. Subsequently, the control layer adds replicas of the index cluster containers, schedules the newly generated containers to the node to be scaled, and starts the index scaling service.

[0092] In some embodiments, the control layer performs a validity check on the expansion request. After receiving the request through a dedicated communication port, the control layer first verifies the integrity of the message (whether it contains key fields such as the request UUID and over-limit details) and the validity of the time; then it verifies the validity of the request source and confirms that the monitoring computing layer node is in the authorized list to prevent illegal operations; finally, it performs a necessary secondary verification. The control layer actively pulls real-time running data from the metadata cluster and cross-compares it with the over-limit status reported by the monitoring computing layer. If it finds that the over-limit has been alleviated, it sends a request to postpone and terminates the process; if it confirms that the over-limit continues, it marks the request as executable.

[0093] In some embodiments, the control layer acquires metadata cluster configuration information and then verifies the integrity of the configuration. Here, the control layer calls the cluster configuration center interface to extract standardized configurations related to expansion, ensuring consistency between the new node and the existing second node. Basic configurations include node tags, operating system images, and container runtime; service configurations cover the index service image version, startup parameters (such as memory cache size, log level, and metadata storage connection address), and resource quotas (such as 4 CPU cores, 8GB memory, and 100GB disk, matching the second node specifications); cluster association configurations include the leader node address, distributed log repository access key, snapshot storage mount path, and inter-node communication encryption certificate. After acquisition, the integrity of the configuration is verified. If any is missing (such as an undefined image version), the missing data is supplemented from the backup data in the configuration center; if the backup is missing, an alarm is triggered and expansion is paused.

[0094] In some embodiments, the control layer instantiates containers based on configuration information and orchestration strategies. Here, the control layer invokes a built-in container orchestration engine (such as Kubernetes) and starts the process in conjunction with configuration information and orchestration strategies. For example, firstly, the host machines to be scaled up are selected, and available host machine nodes in the cluster are scanned. Host machines in the same region as the second node, with the same hardware specifications (CPU / memory / disk matching) and resource utilization below 70% are prioritized, excluding nodes with hardware failures or network anomalies. If there are no available host machines in the same region, a host machine in an adjacent region is selected according to the principle of proximity, ensuring that network latency is ≤50ms. Secondly, a container instance is created, the index service image is pulled on the selected host machine, resource quotas and startup parameters are loaded, the snapshot storage directory is mounted, communication certificates and keys are injected, and health check probes are configured (such as accessing the container / health interface every 5 seconds to check the service status). If the probe fails 3 times consecutively after the container starts (such as the interface returning a 500 error), it is automatically destroyed and rebuilt on another host machine; if the detection is successful, information such as container ID, host machine address, and network port is recorded.

[0095] In some embodiments, the control layer and the first node perform initialization and confirmation processes. Here, after the container instance is ready, the control layer sends a node initialization command to the new node. The new node loads the cluster association configuration, automatically connects to the leader node and log repository, and completes network access and authorization authentication. The control layer assigns a unique node ID to it, with the same format as the second node, marking the first node's role as a new node to be synchronized. After the first node completes initialization, it reports a readiness notification to the control layer, which includes the node ID, resource usage, and service startup status. The control layer verifies the first node's configuration consistency (e.g., whether the image version is consistent with the second node) and service availability (e.g., whether it can receive leader node commands normally). After confirming that everything is correct, the control layer officially defines it as the first node. At this point, the first node has the basic conditions to access the metadata cluster and can proceed to the subsequent snapshot synchronization and log update stages.

[0096] Step 304: Synchronize the information of the first node based on the system status snapshot and operation log entries, so that the synchronized first node is consistent with the system status of the metadata cluster.

[0097] Here, the newly joined first node requests the latest system state snapshot information from the leader node in the cluster. The leader node returns the most recent system state snapshot along with its timestamp. The first node downloads the corresponding snapshot based on the returned information and loads it into memory as the initial data state. Subsequently, the first node begins synchronizing all operation logs from other nodes in the cluster since the snapshot time. After receiving log entries, the new node applies these log entries sequentially, gradually updating its internal state. This process is idempotent, meaning the same log entry can be safely applied multiple times without affecting the final result. When the first node catches up to the latest state of the cluster, it can begin participating in normal services. At this point, the first node continues to receive and apply subsequent log entries to maintain consistency with the cluster.

[0098] In some embodiments, the first node obtains a system state snapshot from the third node, stores the system state snapshot, and constructs the initial system state of the first node based on the system state snapshot.

[0099] Here, after the first node completes initialization, it proactively sends a system state snapshot retrieval request to the leader node of the metadata cluster. This request includes its own node ID and the timestamp to be synchronized, indicating a need to obtain the latest system state snapshot of the cluster. Upon receiving the request, the leader node locates the latest metadata cluster system state snapshot from the cluster snapshot index table. This snapshot integrates the baseline state of all second nodes, including the baseline timestamp, the storage address of each node's snapshot shard, data digest, and other information. The leader node returns this snapshot index information to the first node and simultaneously assigns it a snapshot download authorization token to access the snapshot shards in distributed storage.

[0100] Next, the first node performs parallel download and integrity verification of snapshot fragments. Based on the snapshot index information returned by the leader node, the first node identifies all fragments contained in the cluster snapshot (e.g., 10 fragments split into 1GB fragments) and downloads all fragments in parallel through the distributed storage interface (initiating 3-5 download tasks simultaneously to improve synchronization efficiency). After each fragment is downloaded, the first node immediately calculates the actual hash value of the fragment and compares it with the fragment data digest recorded in the index table: if the hash values ​​match, the fragment is not corrupted and can be stored in the local temporary storage directory; if the hash values ​​do not match (e.g., data loss during transmission), the fragment re-download mechanism is triggered, and the fragment is re-requested from the distributed storage until the complete fragment is downloaded.

[0101] Once all shards have been downloaded and verified, the first node will initiate the snapshot consolidation process. Specifically, the scattered shards are concatenated into a complete cluster system state snapshot according to their shard numbers (e.g., shard 01, shard 02... shard 10). Simultaneously, the metadata information in the snapshot is parsed, including globally unique identifiers for all objects, attributes (size, creation time), the mapping relationship between indexes and storage locations, and the baseline state data of each second node. After consolidation, the first node will batch load the snapshot data into its local in-memory database and mark the snapshot loading as complete. At this point, the first node has an initial system state consistent with the cluster's baseline timestamp.

[0102] In some embodiments, the first node synchronizes operation log entries from the second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster.

[0103] Here, the first node sends a log synchronization request to the system. This request includes a base timestamp of the loaded snapshot, which defines the scope of the operation log synchronization. Only operation log entries with operation timestamps later than the base timestamp need to be synchronized. The first node downloads operation log entries from each second node in batches. After each batch of logs is downloaded, the first node first verifies the log's time order: it checks whether the operation timestamps of all logs within the batch are strictly increasing (if timestamps are out of order, such as log A having a later timestamp than log B but appearing first, it indicates an anomaly in log transmission), and confirms that the start timestamp of this batch of logs is continuous with the end timestamp of the previous batch (to avoid log omissions). If an anomaly is found during verification, the first node requests a resend of the batch of logs; if the verification passes, the log application phase begins.

[0104] In some embodiments, the first node applies the operation log entries one by one to update the initial system state until the first node is consistent with the system state of the metadata cluster.

[0105] Here, the first node applies the changes sequentially based on time, performing corresponding metadata changes on each operation log. If the log type is "ADD" (index addition), the globally unique identifier and attribute information of the object recorded in the log are written to the local in-memory database, and a new index mapping is established. If the log type is "DEL" (index deletion), the corresponding index record is deleted from the local in-memory database according to the ID in the log, and the associated mapping relationship is cleaned up. If the log type is "UPD" (index update), the old data of the corresponding index in the local in-memory database is overwritten according to the change details in the log (such as modifying the storage location of the object, updating the creation time).

[0106] Because each log operation entry contains a unique operation ID and complete change information, this application process is idempotent. Even if a log entry is repeatedly synchronized due to network fluctuations, repeated application will not cause local state errors (e.g., when repeatedly executing a "DEL" operation, if the index has already been deleted, the system will ignore the operation; when repeatedly executing an "ADD" operation, if the index already exists, the system will skip creation). This continues until all incremental logs are synchronized.

[0107] In some embodiments, the first node performs synchronization status verification and consistency confirmation to ensure alignment with the cluster status.

[0108] For example, after the first node completes snapshot loading and all incremental log applications, it calculates a global hash value for all metadata information (including snapshot base data and log incremental data) in its local in-memory database. This hash value is generated by traversing the unique identifiers and core attributes of all object indexes according to a preset algorithm, and is used to reflect the complete data state of the current first node. The first node sends the calculated local global hash value to the system, requesting a state consistency check. The system randomly selects 2-3 healthy second nodes from the metadata cluster, obtains the current global hash value of each node (the second node maintains its own data state hash value in real time), and calculates the average of these hash values. Subsequently, the system returns the cluster's baseline hash value (i.e., the consistent hash value of the second nodes) to the first node.

[0109] The first node compares its local global hash value with the cluster baseline hash value. If they match, it means the first node's local state is fully synchronized with the metadata cluster, and the system marks the first node as available, allowing it to join the cluster's service queue and begin receiving client requests for index queries, additions, etc. If they do not match, the system further locates the differences. By comparing the shard hash values ​​of the first node and the second node, the metadata is split into multiple shards according to the object ID range, and hashes are calculated for each shard. The system then identifies the shards with discrepancies and guides the first node to resynchronize the snapshots and logs corresponding to those shards until the local hash value matches the cluster baseline hash value. At this point, the first node completes information synchronization and officially becomes an effective component of the metadata cluster, sharing the index service with other nodes and maintaining the consistency of the overall cluster data state.

[0110] In some embodiments, the system further includes an Elastic Load Balancer (SLB) server connecting the gateway and nodes. Once the first node completes scaling and successfully joins the cluster, the system automatically adjusts the SLB's backend address configuration and weight, registering the first node to the virtualization server endpoint backend. This process does not require restarting the gateway component, achieving hot updating of the first node's address. After the first node initializes and synchronizes its data, the SLB detects the presence of the new node in the cluster and automatically updates its backend server list. The SLB dynamically adjusts the weight of each node based on its actual load in the current cluster, ensuring that requests are evenly distributed. This allows the first node to immediately begin processing requests without interrupting or affecting existing services. Specifically, refer to... Figure 6 , Figure 3 After step 304 shown, steps 601 to 602 are also executed, which will be explained in detail below.

[0111] Step 601: Detect whether the first node exists in the metadata cluster through the load balancing gateway. If it exists, perform unified access management on the first node to obtain the virtualized service endpoint of the first node.

[0112] In some embodiments, the load balancing gateway can detect the presence of a first node in the metadata cluster by proactively sending a first node ready notification to the load balancing gateway after the first node has completed state synchronization with the cluster. This notification includes the first node's node ID, actual network address (such as IP port), and service status identifier ("ready").

[0113] In some embodiments, the load balancing gateway can proactively and periodically detect the presence of a first node in the metadata cluster. Specifically, the load balancing gateway initiates a "node list query request" to the leader node of the metadata cluster at a preset period to obtain a list of all nodes in the current "available state" (including the second node and the ready first node) in the cluster, and compares it with the historical node list cached locally by the gateway. If a newly added available state node is found and its node ID is not in the historical list, it can be determined that a new first node exists.

[0114] In some embodiments, during the detection process, the load balancing gateway can perform a validity check on the first node information in the first node readiness notification or node list query results. Specifically, the load balancing gateway confirms that the actual network address format of the first node is compliant and the service status identifier is correct to exclude nodes that have not completed synchronization. If the verification fails, the first node is not considered to be accessible, and the abnormal node information is reported, waiting for the node status to be resynchronized.

[0115] In some embodiments, if the detection confirms the existence of a first accessible node, the load balancing gateway will initiate a unified access management process.

[0116] For example, the load balancing gateway standardizes service endpoints by generating corresponding virtualization identifiers based on the actual network address of the first node according to the cluster's preset virtualization service endpoint rules. The load balancing gateway performs health check configurations, binding health probe policies to the first node's virtualization service endpoints, including but not limited to TCP port probes and application layer probes. If consecutive probes fail, the endpoint is automatically marked as abnormal to prevent requests from being forwarded to the faulty node. The load balancing gateway performs access permission authentication by obtaining the first node's service authentication key from the metadata cluster. This key is allocated by the control layer during the first node's initialization. The gateway verifies the key with the first node using an encrypted communication protocol to confirm that the first node is a legitimate node in the cluster, preventing unauthorized nodes from accessing the network. After successful verification, the virtualization service endpoint registration is completed, at which point the first node's virtualization service endpoint has the basic conditions to receive and forward requests.

[0117] Step 602: Update the backend server list based on the virtualization service endpoint of the first node. The backend server list includes the virtualization service endpoints of each node in the metadata cluster.

[0118] In some embodiments, updating the backend server list based on the virtualization service endpoint of the first node includes: obtaining the load status of each node in the metadata cluster through a load balancing gateway, and performing weight adjustment processing on the cluster node list based on the load status of each node and the virtualization service endpoint of the first node to obtain the updated backend service address list.

[0119] Here, before updating the backend server list, the load balancing gateway first obtains real-time load data for all nodes in the cluster (including the second node and the first node that has already been connected) to ensure the rationality of the weight adjustment. The real-time load data includes, but is not limited to, the following load data collection dimensions: CPU utilization, memory utilization, current active connections, requests processed per second, and request response latency for each node. This data is reported locally by the nodes to the gateway's load statistics module at regular intervals.

[0120] In some embodiments, after receiving load data, the load balancing gateway can verify the integrity and timeliness of the data. If the load data of a node is missing or expired, it will actively probe to obtain the latest data. If the probe fails continuously, the node will be marked as having unknown load and temporarily processed with the lowest weight (such as weight value 1).

[0121] In some embodiments, the load balancing gateway can adopt a dynamic weight allocation strategy of "higher load, lower weight," adjusting the weight of the backend server list based on the virtualized service endpoint of the first node and the load of each node. Specifically, load metrics are normalized, converting metrics such as CPU utilization, memory usage, queries per second (QPS), and response latency of each node into a comprehensive load score according to preset weights. Then, weight values ​​are calculated, and a basic weight range for cluster nodes is set. The comprehensive load score and weight value are negatively correlated. Weight conflict handling can also be performed. If multiple nodes calculate the same weight value, the service stability metrics of the nodes are further compared to ensure that the weight allocation is more in line with the actual service capacity of the nodes. At the same time, to avoid excessive request concentration due to excessively high weight of a single node, an upper limit can be set for the weight value, and the excess is distributed proportionally to other low-load nodes.

[0122] In some embodiments, after the load balancing gateway obtains the weights, it restructures the list structure. The new backend server list integrates all node information with "virtualization service endpoint - weight value - health status" as the core field, and deletes the virtualization service endpoints of nodes that have been marked as abnormal (such as continuous health check failures) from the list.

[0123] In some embodiments, the load balancing gateway employs a hot update mechanism, writing the reorganized list of backend servers into a specified field in the gateway configuration file (without restarting the gateway process), and simultaneously notifying the request forwarding module within the gateway to load the new list. The forwarding module completes the switch between the old and new lists within 1 second. During the switch, requests being processed continue to be forwarded using the old list, while newly received requests are immediately allocated according to weight using the new list. After the new list takes effect, the gateway checks whether it forwards requests to the corresponding nodes according to the weight ratio. If a deviation in the forwarding ratio is found, the weight calculation logic and list loading status are re-verified to ensure that the updated backend server list can correctly guide request distribution, ultimately achieving hot updates of index node addresses and balanced distribution of cluster load.

[0124] As described above, this application proposes a method for scaling up a metadata cluster. This method automatically determines whether scaling up is needed based on the current cluster's running data and issues scaling up commands through the control layer to automatically implement the scaling up process. In this method, the monitoring computing layer collects the running status data of the index cluster service, calculates whether the performance boundaries of the cluster design are met, and if the boundaries are exceeded, a scaling up command is issued to the control layer to begin scaling up. Newly added index service nodes are initialized by synchronizing data with existing nodes. This mechanism allows new nodes to obtain only the changed data since the last snapshot, rather than the entire dataset, thus significantly shortening the initialization time. By using an elastic load balancer (SLB) to connect the gateway and the index service, network link configuration decoupling is achieved, avoiding service fluctuations caused by restarting the gateway after scaling up, and enabling hot updates of index node addresses. The observation panel component calls the index cluster status interface to visualize various cluster information and the scaling up process, achieving complete visualization of the cluster's running status and the scaling up process.

[0125] Reference Figure 7 , Figure 7 This is a flowchart illustrating the capacity expansion method provided in this application embodiment, applied to the first node, and combined with... Figure 7 The steps shown are explained.

[0126] Step 701: Obtain a system status snapshot. The system status snapshot includes a timestamp and the metadata information corresponding to the timestamp.

[0127] In some embodiments, the metadata cluster further includes a third node, which is determined by the second node in the metadata cluster based on the first protocol. The third node is used to manage information synchronization between the first node and the second node. Obtaining a system state snapshot includes obtaining a system state snapshot from the third node.

[0128] Here, the third node is the leader node in the metadata cluster, also known as the seed node. The first protocol includes the Raft protocol, which ensures data consistency and high system availability. The third node only commits an update after a majority of nodes have acknowledged receiving and persisting it, thus preventing data loss and inconsistency. If the current third node fails, the Raft protocol automatically triggers a new election to select a new leader to take over log replication and information synchronization management tasks. This incremental synchronization mechanism not only improves the initialization efficiency of new nodes but also enhances the fault tolerance and scalability of the entire system. In this way, the object storage system can smoothly scale without affecting existing services while maintaining data consistency and reliability.

[0129] Here, after the first node completes initialization, it proactively sends a system state snapshot retrieval request to the leader node of the metadata cluster. This request includes its own node ID and the timestamp to be synchronized, indicating a need to obtain the latest system state snapshot of the cluster. Upon receiving the request, the leader node locates the latest metadata cluster system state snapshot from the cluster snapshot index table. This snapshot integrates the baseline state of all second nodes, including the baseline timestamp, the storage address of each node's snapshot shard, data digest, and other information. The leader node returns this snapshot index information to the first node and simultaneously assigns it a snapshot download authorization token to access the snapshot shards in distributed storage.

[0130] Step 702: Store a system state snapshot and construct the initial system state of the first node based on the system state snapshot.

[0131] Here, the first node performs parallel downloading and integrity verification of snapshot fragments. Based on the snapshot index information returned by the leader node, the first node identifies all fragments contained in the cluster snapshot (e.g., 10 fragments split into 1GB fragments) and downloads all fragments in parallel through the distributed storage interface (initiating 3-5 download tasks simultaneously to improve synchronization efficiency). After each fragment is downloaded, the first node immediately calculates the actual hash value of the fragment and compares it with the fragment data digest recorded in the index table: if the hash values ​​match, it means the fragment is not corrupted and can be stored in the local temporary storage directory; if the hash values ​​do not match (e.g., data loss during transmission), the fragment re-download mechanism is triggered, and the fragment is re-requested from the distributed storage until the complete fragment is downloaded.

[0132] Once all shards have been downloaded and verified, the first node will initiate the snapshot consolidation process. Specifically, the scattered shards are concatenated into a complete cluster system state snapshot according to their shard numbers (e.g., shard 01, shard 02... shard 10). Simultaneously, the metadata information in the snapshot is parsed, including globally unique identifiers for all objects, attributes (size, creation time), the mapping relationship between indexes and storage locations, and the baseline state data of each second node. After consolidation, the first node will batch load the snapshot data into its local in-memory database and mark the snapshot loading as complete. At this point, the first node has an initial system state consistent with the cluster's baseline timestamp.

[0133] Step 703: Synchronize operation log entries from the second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster.

[0134] Here, the first node sends a log synchronization request to the system. This request includes a base timestamp of the loaded snapshot, which defines the scope of the operation log synchronization. Only operation log entries with operation timestamps later than the base timestamp need to be synchronized. The first node downloads operation log entries from each second node in batches. After each batch of logs is downloaded, the first node first verifies the log's time order: it checks whether the operation timestamps of all logs within the batch are strictly increasing (if timestamps are out of order, such as log A having a later timestamp than log B but appearing first, it indicates an anomaly in log transmission), and confirms that the start timestamp of this batch of logs is continuous with the end timestamp of the previous batch (to avoid log omissions). If an anomaly is found during verification, the first node requests a resend of the batch of logs; if the verification passes, the log application phase begins.

[0135] Step 704: Apply operation log entries one by one to update the initial system state until the first node is consistent with the system state of the metadata cluster.

[0136] Here, the first node applies the changes sequentially, performing corresponding metadata changes on each operation log entry. If the log type is "ADD" (index addition), the globally unique identifier and attribute information of the object recorded in the log are written to the local in-memory database, establishing a new index mapping. If the log type is "DEL" (index deletion), the corresponding index record is deleted from the local in-memory database based on the ID in the log, and the associated mapping relationship is cleaned up. If the log type is "UPD" (index update), the old data of the corresponding index in the local in-memory database is overwritten according to the change details in the log (such as modifying the object's storage location or updating the creation time).

[0137] Because each log operation entry contains a unique operation ID and complete change information, this application process is idempotent. Even if a log entry is repeatedly synchronized due to network fluctuations, repeated application will not cause local state errors (e.g., when repeatedly executing a "DEL" operation, if the index has already been deleted, the system will ignore the operation; when repeatedly executing an "ADD" operation, if the index already exists, the system will skip creation). This continues until all incremental logs are synchronized.

[0138] In some embodiments, the first node performs synchronization status verification and consistency confirmation to ensure alignment with the cluster status.

[0139] For example, after the first node completes snapshot loading and all incremental log applications, it calculates a global hash value for all metadata information (including snapshot base data and log incremental data) in its local in-memory database. This hash value is generated by traversing the unique identifiers and core attributes of all object indexes according to a preset algorithm, and is used to reflect the complete data state of the current first node. The first node sends the calculated local global hash value to the system, requesting a state consistency check. The system randomly selects 2-3 healthy second nodes from the metadata cluster, obtains the current global hash value of each node (the second node maintains its own data state hash value in real time), and calculates the average of these hash values. Subsequently, the system returns the cluster's baseline hash value (i.e., the consistent hash value of the second nodes) to the first node.

[0140] The first node compares its local global hash value with the cluster baseline hash value. If they match, it means the first node's local state is fully synchronized with the metadata cluster, and the system marks the first node as available, allowing it to join the cluster's service queue and begin receiving client requests for index queries, additions, etc. If they do not match, the system further locates the differences. By comparing the shard hash values ​​of the first node and the second node, the metadata is split into multiple shards according to the object ID range, and hashes are calculated for each shard. The system then identifies the shards with discrepancies and guides the first node to resynchronize the snapshots and logs corresponding to those shards until the local hash value matches the cluster baseline hash value. At this point, the first node completes information synchronization and officially becomes an effective component of the metadata cluster, sharing the index service with other nodes and maintaining the consistency of the overall cluster data state.

[0141] In some embodiments, the control layer and the first node perform initialization and confirmation processes. Here, after the container instance is ready, the control layer sends a node initialization command to the new node. The new node loads the cluster association configuration, automatically connects to the leader node and log repository, and completes network access and authorization authentication. The control layer assigns a unique node ID to it, with the same format as the second node, marking the first node's role as a new node to be synchronized. After the first node completes initialization, it reports a readiness notification to the control layer, which includes the node ID, resource usage, and service startup status. The control layer verifies the first node's configuration consistency (e.g., whether the image version is consistent with the second node) and service availability (e.g., whether it can receive leader node commands normally). After confirming that everything is correct, the control layer officially defines it as the first node. At this point, the first node has the basic conditions to access the metadata cluster and can proceed to the subsequent snapshot synchronization and log update stages.

[0142] As described above, this application proposes a method for scaling up a metadata cluster. This method automatically determines whether scaling up is needed based on the current cluster's running data and issues scaling up commands through the control layer to automatically implement the scaling up process. In this method, the computing layer collects the running status data of the index cluster service, calculates whether the performance boundaries of the cluster design are met, and if the boundaries are exceeded, an scaling up command is issued to the control layer to begin scaling up. Newly added index service nodes are initialized by synchronizing data with existing nodes. This mechanism allows new nodes to obtain only the changed data since the last snapshot, rather than the entire dataset, thus significantly shortening the initialization time. By using an elastic load balancer (SLB) to connect the gateway and the index service, network link configuration decoupling is achieved, avoiding service fluctuations caused by restarting the gateway after scaling up, and enabling hot updates of index node addresses. The observation panel component in the computing layer calls the index cluster status interface to visualize various cluster information and the scaling up process, achieving complete visualization of the cluster's running status and the scaling up process.

[0143] Figure 8 This is a schematic diagram of the structural composition of the metadata cluster expansion device provided in the embodiments of this application. Figure 1 It is used in storage service systems, such as Figure 8 As shown, the metadata cluster expansion device 800 includes:

[0144] The data storage unit 801 is used to periodically store the metadata information of the metadata cluster to obtain a system state snapshot of the metadata cluster.

[0145] The data acquisition unit 802 is used to acquire operation log entries, which include data change operation records sorted by time after storing the system state snapshot.

[0146] The data processing unit 803 is used to determine whether the metadata cluster needs to be expanded based on the running data of the metadata cluster; if expansion is required, a first node is determined, which is a node to be added to the metadata cluster; the first node is used to synchronize information based on the system status snapshot and the operation log entries, so that the synchronized first node is consistent with the system status of the metadata cluster.

[0147] In some embodiments, the metadata cluster includes a second node, which is an existing node in the metadata cluster; the data acquisition unit 802 is further configured to acquire a system status snapshot generated by each second node based on a preset period, the system status snapshot including a timestamp and metadata information corresponding to the timestamp; the data storage unit 801 is further configured to store the system status snapshot of each second node to obtain a system status snapshot of the metadata cluster.

[0148] In some embodiments, the data processing unit 803 is further configured to monitor the running data of the metadata cluster through the computing layer, determine whether the running data is greater than a first threshold based on the running data, and if it is greater, determine that the metadata cluster needs to be expanded and send an expansion request to the control layer; it is also configured to receive the expansion request sent by the computing layer through the control layer, obtain configuration information from the metadata cluster, and perform instantiation processing based on the configuration information through a container orchestration strategy to obtain the first node.

[0149] In some embodiments, the metadata cluster further includes a third node, which is determined by the metadata cluster based on a first protocol, and the third node is used to manage information synchronization between the first node and the second node.

[0150] In some embodiments, the data processing unit 803 is further configured to detect whether a first node exists in the metadata cluster through a load balancing gateway; if it exists, perform unified access management processing on the first node to obtain the virtualization service endpoint of the first node; and update the backend server list based on the virtualization service endpoint of the first node, wherein the backend server list includes the virtualization service endpoints of each node in the metadata cluster.

[0151] In some embodiments, the data processing unit 803 is further configured to obtain the load status of each node in the metadata cluster through the load balancing gateway, and perform weight adjustment processing on the cluster node list based on the load status of each node and the virtualization service endpoint of the first node to obtain an updated backend service address list.

[0152] Those skilled in the art should understand that Figure 8The functionality of each unit in the metadata cluster expansion device shown can be understood by referring to the relevant descriptions of the aforementioned methods. Figure 8 The functions of each unit in the metadata cluster expansion device shown can be implemented by a program running on a processor or by specific logic circuits.

[0153] Figure 9 This is a schematic diagram of the structural composition of the metadata cluster expansion device provided in the embodiments of this application. Figure 2 Applied to the first node, such as Figure 9 As shown, the metadata cluster expansion device 900 includes:

[0154] The data acquisition unit 901 is used to acquire a system status snapshot, which includes a timestamp and metadata information corresponding to the timestamp.

[0155] Data storage unit 902 is used to store the system state snapshot and construct the initial system state of the first node based on the system state snapshot.

[0156] The data synchronization unit 903 is used to synchronize operation log entries from a second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster; the operation log entries are applied one by one based on the time sorting to update the initial system state until the system state of the first node is consistent with that of the metadata cluster.

[0157] In some embodiments, the metadata cluster further includes a third node, which is determined by the second node in the metadata cluster based on a first protocol. The third node is used to manage information synchronization between the first node and the second node. The data acquisition unit 901 is also used to acquire the system status snapshot from the third node.

[0158] Those skilled in the art should understand that Figure 9 The functionality of each unit in the metadata cluster expansion device shown can be understood by referring to the relevant descriptions of the aforementioned methods. Figure 9 The functions of each unit in the metadata cluster expansion device shown can be implemented by a program running on a processor or by specific logic circuits.

[0159] Figure 10 This is a schematic structural diagram of an electronic device 1000 provided in an embodiment of this application. The electronic device may be a server. Figure 10 The illustrated electronic device 1000 includes a processor 1010, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0160] Optionally, such as Figure 10 As shown, the electronic device 1000 may further include a memory 1020. The processor 1010 can retrieve and run computer programs from the memory 1020 to implement the methods described in the embodiments of this application.

[0161] The memory 1020 can be a separate device independent of the processor 1010, or it can be integrated into the processor 1010.

[0162] Optionally, such as Figure 10 As shown, the electronic device 1000 may also include a transceiver 1030, and the processor 1010 may control the transceiver 1030 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices.

[0163] The transceiver 1030 may include a transmitter and a receiver. The transceiver 1030 may further include an antenna, and the number of antennas may be one or more.

[0164] Optionally, the electronic device 1000 may specifically be a server cluster in the embodiments of this application, and the electronic device 1000 may implement the corresponding processes implemented by the server cluster in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0165] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0166] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0167] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0168] This application also provides a computer-readable storage medium for storing computer programs.

[0169] Optionally, the computer-readable storage medium can be applied to the network device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the network device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0170] Optionally, the computer-readable storage medium can be applied to the mobile terminal / terminal device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the mobile terminal / terminal device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0171] This application also provides a computer program product, including computer program instructions.

[0172] Optionally, the computer program product can be applied to the network device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the network device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0173] Optionally, the computer program product can be applied to the mobile terminal / terminal device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the mobile terminal / terminal device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0174] This application also provides a computer program.

[0175] Optionally, the computer program can be applied to the network device in the embodiments of this application. When the computer program is run on the computer, it causes the computer to execute the corresponding processes implemented by the network device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0176] Optionally, the computer program can be applied to the mobile terminal / terminal device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the mobile terminal / terminal device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0177] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for scaling up a metadata cluster, characterized in that, The method includes: The metadata information of the metadata cluster is periodically stored to obtain a system state snapshot of the metadata cluster; Obtain operation log entries, which include data change operation records sorted by time after storing the system state snapshot; Based on the operational data of the metadata cluster, it is determined whether the metadata cluster needs to be expanded; if expansion is required, a first node is identified, which is the node to be added to the metadata cluster; wherein, the operational data is multi-dimensional data collected in real time from all second nodes and associated components of the metadata cluster by the computing layer; the computing layer is used to send an expansion request to the control layer when the metadata cluster is expanded, and the control layer is used to determine the first node based on configuration information and orchestration strategy. Based on the system status snapshot and the operation log entries, the first node is synchronized to ensure that the synchronized first node is consistent with the system status of the metadata cluster. The load balancing gateway detects whether the first node exists in the metadata cluster. If it exists, the first node is processed for unified access management to obtain the virtualized service endpoint of the first node. The backend server list is updated based on the virtualization service endpoint of the first node, and the backend server list includes the virtualization service endpoints of each node in the metadata cluster.

2. The method according to claim 1, characterized in that, The metadata cluster includes a second node, which is an existing node in the metadata cluster; The periodic storage processing of metadata information in the metadata cluster to obtain a system state snapshot of the metadata cluster includes: Obtain a system status snapshot generated by each second node based on a preset period. The system status snapshot includes a timestamp and metadata information corresponding to the timestamp. Store a system state snapshot of each second node to obtain a system state snapshot of the metadata cluster.

3. The method according to claim 2, characterized in that, The step of determining whether to expand the metadata cluster based on the operational data of the metadata cluster includes: The computing layer monitors the operating data of the metadata cluster and determines whether the operating data exceeds a first threshold. If it does, the metadata cluster needs to be expanded, and an expansion request is sent to the control layer.

4. The method according to claim 3, characterized in that, Determining the first node includes: The control layer receives the scaling request sent by the computing layer, obtains configuration information from the metadata cluster, performs instantiation processing based on the configuration information using a container orchestration strategy, and determines the first node.

5. The method according to claim 2, characterized in that, The metadata cluster also includes a third node, which is determined by the metadata cluster based on the first protocol. The third node is used to manage the information synchronization between the first node and the second node.

6. The method according to claim 1, characterized in that, The update of the backend server list based on the virtualization service endpoint of the first node includes: The load balancing gateway obtains the load status of each node in the metadata cluster, and performs weight adjustment processing on the cluster node list based on the load status of each node and the virtualization service endpoint of the first node to obtain an updated backend service address list.

7. A method for scaling up a metadata cluster, characterized in that, Applied to the first node, the method includes: Obtain a system status snapshot, the system status snapshot including a timestamp and metadata information corresponding to the timestamp; Store the system state snapshot and construct the initial system state of the first node based on the system state snapshot; Synchronize operation log entries from a second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster; The operation log entries are applied one by one to update the initial system state until the system state of the first node is consistent with that of the metadata cluster; The first node is determined by the control layer based on configuration information and orchestration strategy after receiving the expansion request sent by the computing layer; the computing layer is used to collect multi-dimensional data from all the second nodes and related components of the metadata cluster in real time. The metadata cluster includes a load balancing gateway, which is used to detect whether the first node exists in the metadata cluster. If it exists, the first node is subjected to unified access management processing to obtain the virtualization service endpoint of the first node. The backend server list is updated based on the virtualization service endpoint of the first node, and the backend server list includes the virtualization service endpoints of each node in the metadata cluster.

8. The method according to claim 7, characterized in that, The metadata cluster also includes a third node, which is determined by the second node in the metadata cluster based on the first protocol. The third node is used to manage the information synchronization between the first node and the second node. The process of obtaining a system status snapshot includes: Obtain the system state snapshot from the third node.

9. A device for expanding the capacity of a metadata cluster, characterized in that, The device, applied to a metadata cluster, includes: A data storage unit is used to periodically store the metadata information of the metadata cluster to obtain a system state snapshot of the metadata cluster. The data acquisition unit is used to acquire operation log entries, which include data change operation records sorted by time after storing the system state snapshot; A data processing unit is used to determine whether the metadata cluster needs to be expanded based on the operational data of the metadata cluster; if expansion is required, a first node is identified, which is a node to be added to the metadata cluster; the first node is synchronized with the system status snapshot and the operation log entries to ensure that the synchronized first node is consistent with the system status of the metadata cluster; wherein, the operational data is multi-dimensional data collected in real time from all second nodes and associated components of the metadata cluster by the computing layer; the computing layer is used to send an expansion request to the control layer when the metadata cluster is expanded, and the control layer is used to determine the first node based on configuration information and orchestration strategy. The data processing unit is further configured to detect whether the first node exists in the metadata cluster through the load balancing gateway; if it exists, perform unified access management processing on the first node to obtain the virtualization service endpoint of the first node; and update the backend server list based on the virtualization service endpoint of the first node, wherein the backend server list includes the virtualization service endpoints of each node in the metadata cluster.

10. A device for expanding the capacity of a metadata cluster, characterized in that, Applied to the first node, the device includes: A data acquisition unit is used to acquire a system status snapshot, the system status snapshot including a timestamp and metadata information corresponding to the timestamp; A data storage unit is used to store the system state snapshot and construct the initial system state of the first node based on the system state snapshot; A data synchronization unit is used to synchronize operation log entries from a second node in the metadata cluster, wherein the operation log entries are data change operation records sorted by time; the second node is an existing node in the metadata cluster; the operation log entries are applied one by one to update the initial system state until the system state of the first node is consistent with that of the metadata cluster; The first node is determined by the control layer based on configuration information and orchestration strategy after receiving the expansion request sent by the computing layer; the computing layer is used to collect multi-dimensional data from all the second nodes and related components of the metadata cluster in real time. The metadata cluster includes a load balancing gateway, which is used to detect whether the first node exists in the metadata cluster. If it exists, the first node is subjected to unified access management processing to obtain the virtualization service endpoint of the first node. The backend server list is updated based on the virtualization service endpoint of the first node, and the backend server list includes the virtualization service endpoints of each node in the metadata cluster.

11. An electronic device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 8.

13. A computer program product, characterized in that, It includes computer program instructions that cause a computer to perform the method as described in any one of claims 1 to 8.

Images