A method and system for data load balancing in a data cluster

Abstract

The application discloses a data load balancing method and system in a data cluster. The method comprises the following steps: organizing the stored data into unit data blocks in the data cluster, and constructing a global data register; when a virtual system needs to acquire data, locating the nearest node according to the data register and acquiring the data, synchronously updating the'releasable' state of all related data copies, and ensuring that at least one complete data copy is unreleasable; in the data arrangement stage, releasing the redundant data copies according to the state flag, and updating the data register. The application realizes global data position perception and state synchronization through the data register, each node will tend to mainly store the data frequently accessed by the local virtual system, thereby forming a data near-task distribution pattern, and based on the state marking mechanism, the data distribution is dynamically optimized, the redundant storage space is recycled, the data access efficiency and the storage load are automatically balanced under the premise of ensuring data integrity and high availability.

Description

Technical Field

[0001] This application belongs to the field of data cluster technology, and specifically relates to a data load balancing method and system in a data cluster. Background Technology

[0002] With the widespread adoption of cloud computing, big data, and artificial intelligence, data clusters composed of numerous physical server nodes have become the core infrastructure for processing massive amounts of data. In such clusters, data is typically stored in a distributed manner, while computing tasks are handled by virtual systems (such as virtual machines and containers) running on various nodes of the cluster. Efficiently managing data storage locations, optimizing data access performance, and achieving overall cluster load balancing are key challenges for improving system efficiency and reducing operating costs. Currently, common industry solutions mainly fall into the following categories, but all of them have certain limitations:

[0003] (1) Static or semi-static data sharding and replication strategy. This is the most basic and widely used technology. When data is written, it is allocated to fixed nodes according to predefined rules (such as hash, range), and a fixed number of replicas (usually 3 replicas) are created and stored on other nodes to achieve fault tolerance. The computing tasks are then scheduled to be executed on the appropriate nodes. (2) Cache-based dynamic acceleration strategy. In order to alleviate the latency of remote data access, a caching layer (such as Redis, Memcached, or local SSD cache) is often introduced at the local or nearby location of the computing node. Hot data is cached in a location closer to the computing. (3) Load balancing strategy based on data migration or rebalancing. Some advanced distributed storage systems (such as Ceph, some HDFS optimization schemes) have data rebalancing functions. When the storage space or access load between nodes is detected to be unbalanced, data blocks will be migrated in the background. (4) Strategies for migrating computing to data. Some frameworks (such as the idea of ​​MapReduce) emphasize "moving computing to where the data is", that is, scheduling computing tasks to be executed on the nodes that store the data they need. However, the above solutions still have defects. In scheme (1), when a computing task is scheduled to a node that does not store the data it needs, it must pull the data from the remote node through the network, resulting in significant network latency and bandwidth consumption, which is a bottleneck for data-intensive computing; at the same time, its storage load is uneven and the replica management efficiency is low. In scheme (2), the cache capacity is limited, resources are insufficient, and there is also a consistency problem. In scheme (3), such migration usually aims at balancing simple storage space or node I / O load, rather than directly aiming at minimizing the data access latency of a specific computing task; the migrated data may still not be co-located with the computing task that will consume it in the future, the migration decision is based on macro cluster indicators, the old replica lacks an effective management mechanism, and active data and stagnant data are not effectively distinguished. In scheme (4), the reverse migration method ignores the multi-consumer scenario of data, fixes the computing on a data node, which not only increases the coupling, but also cannot meet the low latency requirements of all consumers because a piece of data may be frequently accessed by computing tasks in multiple different locations.

[0004] Therefore, there is an urgent need for a new load balancing method that can proactively and dynamically bring data toward computing and intelligently manage the lifecycle of data replicas.

[0005] The information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a data load balancing method and system in a data cluster.

[0007] Specifically, this application claims a data load balancing method in a data cluster, the method comprising the following steps:

[0008] S100, cluster the stored data in the data cluster, organize the clustered data into unit data blocks of fixed size, set a status flag for each unit data block, the status flag at least includes a "releaseable" status; construct a data registry based on the clustered data organization.

[0009] S110, when one of the multiple virtual systems in the data cluster needs to obtain data to be processed of a specific cluster type, the nearest node of the data to be processed is searched and determined according to the data registry; if the nearest node and the physical location where the virtual system is running belong to the same cluster device node, the data to be processed is directly read based on the data address; otherwise, the data to be processed is migrated with the nearest node as the source address and the physical location where the virtual system is running as the target address.

[0010] S120, set the "releaseable" status of each unit data block of the data to be processed in the target address to 0 to indicate that it is in a non-releaseable state, and set the "releaseable" status of each unit data block of the data to be processed in other nodes to 1 according to the data registry to make it in a releaseable state; update the data registry.

[0011] S130: When performing data cleanup in the data cluster, release and delete the unit data blocks with the status flag "Releasable" set to 1, and update the data registry.

[0012] Furthermore, the data structure architecture of the data registry includes: a cluster management layer, used to maintain the basic information of the data registry itself; a metadata service layer, used to maintain metadata unit data block information, provide index information, status update management and access control; and an interface layer, used to provide external service interfaces.

[0013] Furthermore, the data registry is stored at a specific address in the data cluster. When one of the multiple virtual systems in the data cluster obtains data to be processed of a specific cluster type, it first retrieves the data registry cache from the local node cache. If the cache is not found, it retrieves the data registry from the specific address and imports it into the local cache. The data retrieval request of the virtual system is preferentially sent to the data registry sub-segment. If the data registry sub-segment contains information related to the data to be processed, the target data is obtained based on the information related to the data to be processed. When the data retrieval request cannot be responded to correctly in the sub-segment, the virtual system redirects the data retrieval request to the specific address in the data cluster to obtain the complete data registry.

[0014] Further, step S120 includes:

[0015] S121. Determine the "releaseable" status value of each unit data block of the data to be processed in the source address; if it is 0, proceed to step S122; if it is 1, proceed to step S123.

[0016] S122. Set the "Releaseable" status value of the unit data block with a "Releaseable" status value of 0 to 1, and at the same time set the "Releaseable" status value of the corresponding unit data block copy in the target address to 0.

[0017] S123. Query the data registry to determine other data copies of the unit data block with a "releasable" status value of 1, select the data copy with a "releasable" status value of 0, and set the status value to 1.

[0018] Further, in step S130, when performing data cleanup in the data cluster, releasing and deleting unit data blocks with the status flag "releaseable" set to 1 specifically involves: releasing and deleting unit data blocks with the status flag "releaseable" set to 1 in all storage nodes of the data cluster, and simultaneously migrating and cleanup unit data blocks with the status flag "releaseable" set to 0 stored in different nodes according to the data cleanup requirements; or, releasing and deleting unit data blocks with the status flag "releaseable" set to 1 in the local node, and simultaneously migrating and cleanup unit data blocks with the status flag "releaseable" set to 0 in the local node according to the data cleanup requirements.

[0019] A data load balancing system in a data cluster is also provided, the system comprising:

[0020] The data registry construction module is configured to cluster stored data in the data cluster, organize the clustered data into unit data blocks of fixed size, set a status flag for each unit data block, and the status flag includes at least a "releaseable" status; and construct a data registry based on the clustered data organization.

[0021] The data acquisition module is configured to, when one of the multiple virtual systems in the data cluster needs to acquire data of a specific cluster type to be processed, search and determine the nearest node of the data to be processed according to the data registry; if the nearest node and the physical location where the virtual system is running belong to the same cluster device node, then the data to be processed is directly read based on the data address; otherwise, the data to be processed is migrated with the nearest node as the source address and the physical location where the virtual system is running as the target address.

[0022] The state synchronization module is configured to set the "releaseable" status of each unit data block of the data to be processed in the target address to 0, indicating that it is in a non-releaseable state, and set the "releaseable" status of each unit data block of the data to be processed in other nodes to 1 according to the data registry, so that it is in a releaseable state; and update the data registry.

[0023] The data cleanup module is configured to release and delete unit data blocks with the status flag "releasable" set to 1 and update the data registry when performing data cleanup in the data cluster.

[0024] Furthermore, in the data registry construction module, the data structure architecture of the data registry includes:

[0025] The cluster management layer is used to maintain the basic information of the data registry itself.

[0026] The metadata service layer is used to maintain metadata unit data block information, and provides index information, status update management and access control.

[0027] The interface layer is used to provide external service interfaces.

[0028] Furthermore, the data registry is stored at a specific address in the data cluster; the data acquisition module specifically includes:

[0029] The cache management unit is used to enable the virtual system to first retrieve the data registry cache from the local node cache. If the cache is not found, the data registry is retrieved from the specific address and imported into the local cache.

[0030] The request distribution unit is used to send the data acquisition request of the virtual system to the data registry sub-segment first. If the data registry sub-segment has information related to the data to be processed, the target data is obtained according to the information related to the data to be processed.

[0031] The request redirection unit is used to redirect the data retrieval request to a specific address in the data cluster to obtain the complete data registry when the data retrieval request cannot be responded to correctly in the sub-shard.

[0032] Furthermore, the state synchronization module is configured to perform:

[0033] S121. Determine the "releaseable" status value of each unit data block of the data to be processed in the source address; if it is 0, execute operation S122; if it is 1, execute operation S123.

[0034] S122. Set the "Releaseable" status value of the unit data block with a "Releaseable" status value of 0 to 1, and at the same time set the "Releaseable" status value of the corresponding unit data block copy in the target address to 0.

[0035] S123. Query the data registry to determine other data copies of the unit data block with a "releasable" status value of 1, select the data copy with a "releasable" status value of 0, and set the status value to 1.

[0036] Furthermore, the data processing module is configured to perform any of the following operations when performing data processing:

[0037] Operation 1: Release and delete the unit data blocks with the status flag "Releaseable" set to 1 in all storage nodes in the data cluster. At the same time, according to the data reorganization requirements, migrate and reorganize the unit data blocks with the status flag "Releaseable" set to 0 stored in different nodes.

[0038] Operation 2: Release and delete the unit data blocks in the local node whose status flag is "Releaseable" is 1. At the same time, according to the data cleanup requirements, migrate and clean up the unit data blocks in the local node whose status flag is "Releaseable" is 0.

[0039] This invention provides a data load balancing method and system in a data cluster. The system has at least the following technical effects: (1) It realizes automatic load balancing and efficiency improvement of data storage: Through the "nearest node" access strategy and data migration mechanism, the system can automatically guide data to the vicinity of the physical node where the virtual system that has consumption needs is located. After multiple iterations of data processing migration, each node will tend to store the data frequently accessed by the local virtual system, thus forming a data near task distribution pattern, significantly improving data acquisition efficiency, and achieving load balancing of each node in terms of storage content, avoiding congestion of a single node caused by concentrated data hotspots. (2) By globally synchronizing the "releaseable" status of each unit data block in the data registry, it ensures that at any time, at least one complete data copy with the status of "non-releaseable" is maintained in the cluster, ensuring data integrity; (3) By adopting the strategy of migration first, marking later, and deleting at the opportune time, redundant storage space is intelligently reclaimed under the premise of ensuring data integrity and high availability. (4) Allows the same data to be shared and consumed concurrently by multiple virtual systems. Combined with a multi-level (sub-sharding, local cache) data registry architecture, the system can quickly locate the optimal data source for the virtual system with extremely low metadata query overhead, achieving the optimal balance between data security, access efficiency and storage cost. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only used to illustrate some embodiments of this application. For those skilled in the art, without creative effort, they can obtain other technical features, connection relationships, and even method steps not mentioned in the accompanying drawings based on these drawings.

[0041] Figure 1 This is a flowchart of a data load balancing method in a data cluster provided by an embodiment of the present invention.

[0042] Figure 2 This is a structural diagram of a data load balancing system in a data cluster provided by an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] Example 1: As Figure 1 As shown, the present invention provides a data load balancing method in a data cluster, the method comprising the following steps:

[0047] S100, In the data cluster, the stored data is clustered, and the clustered data is organized into unit data blocks of fixed size. A status flag is set for each unit data block. The status flag includes at least a "releaseable" status. A data registry is constructed according to the organization of the clustered data.

[0048] In a data cluster environment, stored data content is a crucial resource. This data content is typically organized in an orderly manner based on multiple dimensions. For example, data frequently accessed by the same batch of computing tasks can be grouped together, or it can be categorized according to its function (user profile data, product detail data, etc.). Data clustering should possess general rationality; it is an effective way of data organization, but not a fixed or absolute method. In this implementation, when providing data resources externally, data is organized into fixed-size unit data blocks. This is the smallest unit for data management, and its size is determined based on network topology, task content, and other requirements. For example, depending on actual needs, it can be set to 1G, 100M, etc. Management is achieved through status flags. In one embodiment, the status flags at least include a {"Releasable"} status to indicate whether the smallest unit can be released from storage space. In a more preferred embodiment, the status flags also include five-element status information: {("Updating"; "Migrable"; "Readable"); "State Lock"; "Version Value"}.

[0049] Based on the data organization within the data cluster, a data registry is constructed. The data registry is the core data structure used by the entire data cluster to manage data; it is a record table of data storage rules and status, forming a global data map. When the virtual system needs data, it can query this table to find the data's location. In one embodiment, the data registry contains the storage address and current status flag of any unit data block. In a more preferred embodiment, the data registry establishes a multi-level index structure for any unit data block, such as clustering index, node index, timestamp index, status index, hotspot information index, and spatial index, to improve data retrieval and navigation efficiency. In a preferred embodiment, the data structure architecture of the data registry includes: a cluster management layer, used to maintain the basic information of the data registry itself, specifically including naming information (used to automatically name each piece of data), protocol rule information (used to provide protocol rules), version information (used to determine version information), heartbeat update data (used to update data based on heartbeat rules), cache information (used to record cache information), and partition / shard information (used to determine the partition information of the current part of the data registry); a metadata service layer, used to maintain metadata unit data block information, including providing index information, status update management, and access control; and an interface layer, used to provide external service interfaces, including query interfaces, update interfaces, event notifications, event monitoring, and extension interfaces.

[0050] In a preferred embodiment, the data registry is stored at a specific address within the data cluster. When one of the multiple virtual systems in the data cluster needs to retrieve data of a specific cluster type, it first retrieves the data registry cache from its local node cache. If the cache misses or the cached data expires, it retrieves the data registry from the aforementioned specific address and imports it into its local cache. In a preferred embodiment, the data registry can be fragmented and expanded. A virtual system can obtain the complete data registry to further retrieve target data, or it can obtain a sub-fragment of the data registry to retrieve target data based on the data information recorded in the sub-fragment. The virtual system's data retrieval request is preferentially sent to the sub-fragment of the data registry. If the sub-fragment contains relevant information about the target data, the target data is retrieved based on the existing relevant information. When the data retrieval request cannot be responded to correctly in the sub-fragment, the virtual system can redirect the data retrieval request to the specific address in the data cluster to obtain the complete data registry. Setting sub-fragments can reduce the size of the data registry and improve the speed of loading, analysis, and searching.

[0051] S110, when one of the multiple virtual systems in the data cluster needs to obtain data to be processed of a specific cluster type, the nearest node of the data to be processed is searched and determined according to the data registry; if the nearest node and the physical location where the virtual system is running belong to the same cluster device node, the data to be processed is directly read based on the data address; otherwise, the data to be processed is migrated with the nearest node as the source address and the physical location where the virtual system is running as the target address.

[0052] In a data cluster, data is typically stored in a distributed manner across various distributed server nodes (in special cases, it may be concentrated on one or a few server nodes). Computational tasks are carried out by virtual systems (such as virtual machines or containers) running on the server nodes of the data cluster. Data acquisition is the foundation and primary step of any computational task. In this embodiment, based on the relevant information in the data registry and the provided data interface, the data storage status of the data to be processed can be quickly determined, thereby determining the data acquisition strategy.

[0053] The registry contains address information for the data to be processed, such as storage nodes and storage locations. Since the data to be processed may contain multiple copies in the network topology, retrieving the data based on the nearest node can obtain the data at the lowest cost, improving data retrieval efficiency. In one embodiment, the nearest node is determined by determining the physical location of the node, or by determining the node's data transfer rate (fastest). If the nearest node determined according to the data registry belongs to the same cluster device node as the physical location where the virtual system is running, the data to be processed can be directly retrieved locally; otherwise, data migration is required.

[0054] S110 specifically includes the following sub-steps:

[0055] S111: Determine the virtual system requirement type, trigger the data registry retrieval and start the query process;

[0056] S112: Based on the query results, determine the location of all replicas of the data to be processed in the data cluster, and identify the nearest node as the source data node. For example, key considerations include proximity in network topology, lowest latency, sufficient bandwidth, lightest node load, highest node stability, and lowest data transmission cost.

[0057] S113, perform a locality check on the nearest node. If it meets the locality requirement, retrieve the data directly based on the data address; otherwise, continue execution. This step is particularly important in the solution, aiming to reduce the number of data migration operations through rules, and is a key step in ensuring the balanced operation of the entire data cluster. Specifically, it checks several pre-determined situations where data retrieval based on data migration is unnecessary. These situations are considered to meet data locality requirements. For example, data migration is unnecessary when the data is on the same physical machine node. Other acceptable situations should also be considered, such as data on the same server node or the same NUMA node. In these cases, data migration would cause greater overhead and is not the preferred method for data retrieval. Furthermore, in a preferred embodiment, more stringent data migration rules can be defined, such as when the nearest node and the node where the virtual system is running are on the same routing link, within the same network domain, or in a "nearby" network topology. The specific rule-making method can be dynamically adjusted according to actual system requirements and performance requirements.

[0058] S114, Execute migration decisions based on data requirements. This step is the actual execution step of data migration. In practical engineering applications, the data source of the data to be processed may have data replicas at different node locations, and the data source of the data to be processed may consist of multiple different nodes. In these cases, the clustering data adapted to this computing task can be aligned with the computing task through migration operations. Since the computing task types of nodes are usually stable, for example, some computing nodes are suitable for handling model training tasks, while some nodes are suitable for handling logical computing tasks, such migration decisions not only reduce data processing costs but also balance the computing and storage load of the entire cluster. After multiple migrations, the nodes in the data cluster tend to store only the data that their own nodes have consumption needs, thus greatly improving the efficiency of data storage and retrieval and ensuring data load balance.

[0059] S120, set the "releaseable" status of each unit data block of the data to be processed in the target address to 0 to indicate that it is in a non-releaseable state, and set the "releaseable" status of each unit data block of the data to be processed in other nodes to 1 according to the data registry to make it in a releaseable state; update the data registry.

[0060] Based on the principle of protecting new copies and marking old copies, step S120 realizes a switch of the authoritative copy, so that the data cluster always has at least one protected complete data copy, and at the same time provides a basis for marking for safe reclamation of space.

[0061] The "releaseable" status of a data copy of the data to be processed in the data source may be 1, which means that there are other identical data copies in the data cluster with a "releaseable" status of 0. If the migrated data to be processed is simply set to a "releaseable" status of 0, it may result in multiple sets of redundant and unreleaseable data to be processed in the data cluster. Therefore, this situation needs to be handled properly. Specifically, in a preferred embodiment, step S120 includes: S121 determining the "releaseable" status value of each unit data block of the data to be processed in the source address; if it is 0, proceed to step S122, if it is 1, proceed to step S123; S122 setting the "releaseable" status value of the unit data block with a "releaseable" status value of 0 to 1, and simultaneously setting the "releaseable" status value of the corresponding unit data block copy in the target address to 0; S123 querying the data registry for the unit data block with a "releaseable" status value of 1 to determine other data copies of the unit data block with a "releaseable" status value of 1, selecting the data copy with a "releaseable" status value of 0 and setting its status value to 1. This preferred embodiment demonstrates the basic principle of releasable state value flipping and an implementation method based on this principle, which enables the data copy to be securely stored in the node that last uses the data copy. In specific implementations, without departing from the above basic principle, there are also a variety of implementation methods. These inexhaustible implementation methods are easily foreseen by those skilled in the art under the above basic principle. For example, step S123 can be placed as a pre-step to reduce the overhead of accessing, reading, and querying the data registry. Or, the state value flipping operation can be integrated to improve data processing efficiency.

[0062] In a preferred embodiment, consider a scenario where three virtual system computing tasks have overlapping requirements for the data to be processed. The three data analysis tasks start simultaneously and trigger data migration, requiring the generation of the same sales report for the same or partially identical unit data block set S. In this scenario of overlapping requirements, the following steps are performed: (1) Set richer status values ​​for the unit data block, for example, {"Updating", "Readable", "Migration", "Critical Lock", "Releaseable"} ("Updating" indicates that the data in the unit data block is being updated, "Readable" indicates that the data in the unit data block is being read effectively, and "Migration" indicates that the unit data block is in a migration state). "Critical lock" is used to indicate whether a unit data block is in a locked state, and "releaseable" is used to indicate that the unit data block can be removed from the storage space); (2) Query the data registry for all data copies in the current unit data block, determine the unit data block with a "migration in progress" status value of 1, and further determine the task information of the computing task corresponding to the unit data block with a "migration in progress" status value of 1; (3) Compare all the determined task information, select the computing task with the largest task overhead, set the "releaseable" status value of the unit data block involved in the data migration process triggered by it to 0, and set the "releaseable" status value of the corresponding unit data block involved in other computing tasks to 1. The above migration strategy can make the data concentrate as much as possible on the node where the computing task with the large computing overhead is located, thereby balancing the overall load of the data cluster.

[0063] Finally, update the data registry to register the changes in data status resulting from this batch of migration operations for subsequent data queries.

[0064] S130: When performing data cleanup in the data cluster, release and delete the unit data blocks with the status flag "Releasable" set to 1, and update the data registry.

[0065] Data cleanup in a data cluster can be divided into two scenarios. One is data cleanup of the entire data cluster. In this case, all data blocks with the "releaseable" status flag of 1 on all storage nodes in the cluster are released and deleted. Simultaneously, based on the cleanup needs, data blocks with the "releaseable" status flag of 0 stored on different nodes are migrated and cleaned. The other scenario is data cleanup of individual storage nodes within the data cluster. In this case, all data blocks with the "releaseable" status flag of 1 on all storage nodes in the cluster are released and deleted. If the required space is still insufficient, data blocks with the "releaseable" status flag of 0 on that node can be migrated and cleaned to obtain more storage space. Finally, the data registry is updated to record the changes in data status resulting from this batch of data cleanup operations for subsequent data queries.

[0066] Example 2: Figure 2 As shown, the present invention provides a data load balancing system in a data cluster, comprising:

[0067] The data registry construction module is configured to cluster stored data in the data cluster, organize the clustered data into unit data blocks of fixed size, set a status flag for each unit data block, and the status flag includes at least a "releaseable" status; and construct a data registry based on the clustered data organization.

[0068] The data acquisition module is configured to, when one of the multiple virtual systems in the data cluster needs to acquire data of a specific cluster type to be processed, search and determine the nearest node of the data to be processed according to the data registry; if the nearest node and the physical location where the virtual system is running belong to the same cluster device node, then the data to be processed is directly read based on the data address; otherwise, the data to be processed is migrated with the nearest node as the source address and the physical location where the virtual system is running as the target address.

[0069] The state synchronization module is configured to set the "releaseable" status of each unit data block of the data to be processed in the target address to 0, indicating that it is in a non-releaseable state, and set the "releaseable" status of each unit data block of the data to be processed in other nodes to 1 according to the data registry, so that it is in a releaseable state; and update the data registry.

[0070] The data cleanup module is configured to release and delete unit data blocks with the status flag "releasable" set to 1 and update the data registry when performing data cleanup in the data cluster.

[0071] In a preferred embodiment, the data registry construction module includes a data structure architecture comprising:

[0072] The cluster management layer is used to maintain the basic information of the data registry itself.

[0073] The metadata service layer is used to maintain metadata unit data block information, and provides index information, status update management and access control.

[0074] The interface layer is used to provide external service interfaces.

[0075] In a preferred embodiment, the data registry is stored at a specific address in the data cluster; the data acquisition module specifically includes:

[0076] The cache management unit is used to enable the virtual system to first retrieve the data registry cache from the local node cache. If the cache is not found, the data registry is retrieved from the specific address and imported into the local cache.

[0077] The request distribution unit is used to send the data acquisition request of the virtual system to the data registry sub-segment first. If the data registry sub-segment has information related to the data to be processed, the target data is obtained according to the information related to the data to be processed.

[0078] The request redirection unit is used to redirect the data retrieval request to a specific address in the data cluster to obtain the complete data registry when the data retrieval request cannot be responded to correctly in the sub-shard.

[0079] In a preferred embodiment, the state synchronization module is configured to perform:

[0080] S121. Determine the "releaseable" status value of each unit data block of the data to be processed in the source address; if it is 0, execute operation S122; if it is 1, execute operation S123.

[0081] S122. Set the "Releaseable" status value of the unit data block with a "Releaseable" status value of 0 to 1, and at the same time set the "Releaseable" status value of the corresponding unit data block copy in the target address to 0.

[0082] S123. Query the data registry to determine other data copies of the unit data block with a "releasable" status value of 1, select the data copy with a "releasable" status value of 0, and set the status value to 1.

[0083] In a preferred embodiment, the data processing module is configured to perform any of the following operations when performing data processing:

[0084] Operation 1: Release and delete the unit data blocks with the status flag "Releaseable" set to 1 in all storage nodes in the data cluster. At the same time, according to the data reorganization requirements, migrate and reorganize the unit data blocks with the status flag "Releaseable" set to 0 stored in different nodes.

[0085] Operation 2: Release and delete the unit data blocks in the local node whose status flag is "Releaseable" is 1. At the same time, according to the data cleanup requirements, migrate and clean up the unit data blocks in the local node whose status flag is "Releaseable" is 0.

[0086] This invention provides a data load balancing method and system in a data cluster. The system has at least the following technical effects: (1) It realizes automatic load balancing and efficiency improvement of data storage: Through the "nearest node" access strategy and data migration mechanism, the system can automatically guide data to the vicinity of the physical node where the virtual system that has consumption needs is located. After multiple iterations of data processing migration, each node will tend to store the data frequently accessed by the local virtual system, thus forming a data near task distribution pattern, significantly improving data acquisition efficiency, and achieving load balancing of each node in terms of storage content, avoiding congestion of a single node caused by concentrated data hotspots. (2) By globally synchronizing the "releaseable" status of each unit data block in the data registry, it ensures that at any time, at least one complete data copy with the status of "non-releaseable" is maintained in the cluster, ensuring data integrity; (3) By adopting the strategy of migration first, marking later, and deleting at the opportune time, redundant storage space is intelligently reclaimed under the premise of ensuring data integrity and high availability. (4) Allows the same data to be shared and consumed concurrently by multiple virtual systems. Combined with a multi-level (sub-sharding, local cache) data registry architecture, the system can quickly locate the optimal data source for the virtual system with extremely low metadata query overhead, achieving the optimal balance between data security, access efficiency and storage cost.

[0087] According to one embodiment, a program product of a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figure 1 The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0088] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0089] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0090] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this specification should be defined by the appended claims.

[0091] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or they may be jointly implemented by certain components in multiple independent devices.

[0092] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0093] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0094] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles applicable herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A data load balancing method in a data cluster, characterized in that, The method includes the following steps: S100, In the data cluster, the stored data is clustered, and the clustered data is organized into unit data blocks of fixed size. A status flag is set for each unit data block, and the status flag includes at least the "releaseable" status. A data registry is constructed according to the clustered data organization. S110, when one of the multiple virtual systems in the data cluster needs to obtain data to be processed of a specific cluster type, the nearest node of the data to be processed is searched and determined according to the data registry; if the nearest node and the physical location where the virtual system is running belong to the same cluster device node, the data to be processed is directly read based on the data address; otherwise, the data to be processed is migrated with the nearest node as the source address and the physical location where the virtual system is running as the target address. S120, set the "releaseable" status of each unit data block of the data to be processed in the target address to 0 to indicate that it is in a non-releaseable state, and set the "releaseable" status of each unit data block of the data to be processed in other nodes to 1 according to the data registry to make it in a releaseable state; update the data registry. S130, when performing data cleanup in the data cluster, release and delete the unit data blocks with the status flag "releasable" set to 1, and update the data registry; The data registry is stored at a specific address in the data cluster. When one of the multiple virtual systems in the data cluster obtains data of a specific cluster type to be processed, it first retrieves the data registry cache from the local node cache. If the cache is not found, it retrieves the data registry from the specific address and imports it into the local cache. The data retrieval request of the virtual system is preferentially sent to the data registry sub-segment. If the data registry sub-segment contains information related to the data to be processed, the target data is obtained based on the information related to the data to be processed. When the data retrieval request cannot be responded to correctly in the sub-segment, the virtual system redirects the data retrieval request to the specific address in the data cluster to obtain the complete data registry. The status flag also includes five-element status information: {("Updating"; "Migration"; "Readable"); "Status Lock"; "Version Value"}.

2. The data load balancing method in a data cluster as described in claim 1, characterized in that, The data structure architecture of the data registry includes: a cluster management layer, which is used to maintain the basic information of the data registry itself; The metadata service layer is used to maintain metadata unit data block information, provide index information, status update management and access control; the interface layer is used to provide external service interfaces.

3. The data load balancing method in a data cluster as described in claim 1, characterized in that, Step S120 includes: S121, determining the "releaseable" status value of each unit data block of the data to be processed in the source address; if it is 0, proceed to step S122, if it is 1, proceed to step S123; S122, setting the "releaseable" status value of the unit data block with a "releaseable" status value of 0 to 1, and simultaneously setting the "releaseable" status value of the corresponding unit data block copy in the target address to 0; S123, querying the data registry to determine other data copies of the unit data block with a "releaseable" status value of 1, selecting the data copy with a "releaseable" status value of 0 from them and setting its status value to 1.

4. The data load balancing method in a data cluster as described in claim 1, characterized in that, In step S130, when performing data cleanup in the data cluster, releasing and deleting unit data blocks with the status flag "releaseable" set to 1 specifically involves: releasing and deleting unit data blocks with the status flag "releaseable" set to 1 in all storage nodes of the data cluster; simultaneously, migrating and cleanup unit data blocks with the status flag "releaseable" set to 0 stored in different nodes according to the data cleanup requirements; or, releasing and deleting unit data blocks with the status flag "releaseable" set to 1 in the local node; simultaneously, migrating and cleanup unit data blocks with the status flag "releaseable" set to 0 in the local node according to the data cleanup requirements.

5. A data load balancing system in a data cluster, characterized in that, The system includes: A data registry building module is configured to cluster stored data in a data cluster, organize clustered data into unit data blocks of fixed size, set a status flag for each unit data block, and the status flag includes at least a "releaseable" status; and build a data registry based on the clustered data organization. The data acquisition module is configured to, when one of the multiple virtual systems in the data cluster needs to acquire data of a specific cluster type to be processed, search and determine the nearest node of the data to be processed according to the data registry; if the nearest node and the physical location where the virtual system is running belong to the same cluster device node, then the data to be processed is directly read based on the data address; otherwise, the data to be processed is migrated with the nearest node as the source address and the physical location where the virtual system is running as the target address. The state synchronization module is configured to set the "releaseable" status of each unit data block of the data to be processed in the target address to 0, indicating that it is in a non-releaseable state, and set the "releaseable" status of each unit data block of the data to be processed in other nodes to 1 according to the data registry, so that it is in a releaseable state; and update the data registry. The data cleanup module is configured to release and delete unit data blocks with the status flag "releasable" set to 1 and update the data registry when performing data cleanup in the data cluster. The data registry is stored at a specific address in the data cluster. When one of the multiple virtual systems in the data cluster obtains data of a specific cluster type to be processed, it first retrieves the data registry cache from the local node cache. If the cache is not found, it retrieves the data registry from the specific address and imports it into the local cache. The data retrieval request of the virtual system is preferentially sent to the data registry sub-segment. If the data registry sub-segment contains information related to the data to be processed, the target data is obtained based on the information related to the data to be processed. When the data retrieval request cannot be responded to correctly in the sub-segment, the virtual system redirects the data retrieval request to the specific address in the data cluster to obtain the complete data registry. The status flag also includes five-element status information: {("Updating"; "Migration"; "Readable"); "Status Lock"; "Version Value"}.

6. The system as described in claim 5, characterized in that, The data registry construction module includes the following data structure architecture: a cluster management layer for maintaining the basic information of the data registry itself; a metadata service layer for maintaining metadata unit data block information, providing index information, status update management, and access control; and an interface layer for providing external service interfaces.

7. The system as described in claim 5, characterized in that, The state synchronization module is configured to perform the following steps: S121, determine the "releaseable" status value of each unit data block of the data to be processed in the source address; if it is 0, perform operation S122, if it is 1, perform operation S123; S122, set the "releaseable" status value of the unit data block with a "releaseable" status value of 0 to 1, and simultaneously set the "releaseable" status value of the corresponding unit data block copy in the target address to 0; S123, query the data registry to determine other data copies of the unit data block with a "releaseable" status value of 1, select the data copy with a "releaseable" status value of 0 from them, and set its status value to 1.

8. The system as described in claim 5, characterized in that, The data processing module is configured to perform any of the following operations when performing data processing: Operation 1: Release and delete the unit data blocks with the status flag "Releaseable" set to 1 in all storage nodes in the data cluster, and simultaneously migrate and process the unit data blocks with the status flag "Releaseable" set to 0 stored in different nodes according to the data processing requirements; Operation 2: Release and delete the unit data blocks with the status flag "Releaseable" set to 1 in the local node, and simultaneously migrate and process the unit data blocks with the status flag "Releaseable" set to 0 in the local node according to the data processing requirements.

Images