Metadata Management Method and Device

Abstract

This application relates to a metadata management method and apparatus. The method includes: acquiring multi-dimensional attribute information of metadata; generating a multi-dimensional sharding key based on the multi-dimensional attribute information and dynamically adjusted weighting coefficients, and allocating metadata to corresponding storage nodes according to the multi-dimensional sharding key to form metadata shards; wherein the weighting coefficients are used to balance the influence of different dimensional attribute information in the generation of sharding keys; acquiring the real-time load status of each storage node; and triggering a shard migration operation when it is determined that load balancing needs to be performed based on the real-time load status, migrating the metadata shards from the source node to the target node. By introducing multi-dimensional attribute information and dynamically adjusted weighting coefficients to generate sharding keys, the data distribution can be adaptively adjusted according to the real-time load status, avoiding hotspot concentration.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to metadata management methods and apparatus. Background Technology

[0002] With the rapid development of cloud computing, big data, and artificial intelligence technologies, the scale of metadata that distributed storage systems need to process is exploding. As data describing data, the management efficiency of metadata directly determines the performance ceiling of the entire distributed system. In typical distributed file systems or object storage systems, how to rationally allocate massive amounts of metadata to different storage nodes and ensure that the system can provide efficient and stable services has become a key technical challenge in this field.

[0003] In related technologies, distributed metadata management typically employs a static sharding strategy based on hash functions. For example, consistent hashing maps metadata to different storage nodes to achieve basic data distribution and load balancing. However, this type of static mapping method relies heavily on predefined sharding rules, and its rigid design makes it difficult to adapt to dynamically changing data access patterns. When there is a sudden surge in traffic or certain metadata becomes a high-frequency access hotspot, the static sharding strategy cannot adjust the data distribution in real time, resulting in a large number of requests being concentrated on a few nodes, creating a performance bottleneck. Summary of the Invention

[0004] This application provides a metadata management method and apparatus.

[0005] According to a first aspect of the embodiments of this application, a metadata management method is provided, the method comprising: Retrieve multi-dimensional attribute information of metadata; Based on the multi-dimensional attribute information and dynamically adjusted weight coefficients, a multi-dimensional sharding key is generated, and the metadata is allocated to the corresponding storage nodes according to the multi-dimensional sharding key to form metadata shards; wherein, the weight coefficients are used to balance the influence of different dimensional attribute information in the generation of sharding keys; Obtain the real-time load status of each storage node; When it is determined that load balancing needs to be performed based on the real-time load status, a shard migration operation is triggered to migrate the metadata shard from the source node to the target node.

[0006] According to a second aspect of the embodiments of this application, a metadata management apparatus is provided, the apparatus comprising: The information acquisition module is used to acquire multi-dimensional attribute information of metadata; The multidimensional sharding key generation and allocation module is used to generate multidimensional sharding keys based on the multidimensional attribute information and dynamically adjusted weight coefficients, and to allocate the metadata to the corresponding storage nodes according to the multidimensional sharding keys to form metadata shards; wherein, the weight coefficients are used to balance the influence of different dimensional attribute information in the generation of sharding keys; The load status acquisition module is used to acquire the real-time load status of each storage node; The metadata shard migration module is used to trigger a shard migration operation when it is determined that load balancing needs to be performed based on the real-time load status, and to migrate the metadata shards from the source node to the target node.

[0007] According to a third aspect of the embodiments of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described above in this application.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above in this application.

[0010] The metadata management method and apparatus provided in this application introduce multi-dimensional attribute information and dynamically adjusted weight coefficients to generate sharding keys, which can adaptively adjust the data distribution according to the real-time load status and avoid hotspot concentration. At the same time, combined with distributed load monitoring and automated sharding migration mechanism, the system can dynamically balance the load of each node without interrupting service, which significantly improves the scalability, stability and resource utilization of the distributed storage system. Attached Figure Description

[0011] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart illustrating a metadata management method provided in an exemplary embodiment of this application; Figure 2 A schematic diagram of multidimensional fragmentation key generation provided for an exemplary embodiment of this application; Figure 3 A schematic diagram of distributed load monitoring provided for an exemplary embodiment of this application; Figure 4 A schematic diagram of fragment migration provided for an exemplary embodiment of this application; Figure 5A schematic block diagram of the functional modules of a metadata management device provided in an exemplary embodiment of this application; Figure 6 A structural block diagram of an electronic device provided in an exemplary embodiment of this application; Figure 7 A structural block diagram of a computer system provided for an exemplary embodiment of this application. Detailed Implementation

[0012] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0013] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0018] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this application's technical solution, based on the prompt message.

[0019] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of this application; other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0020] This application first provides a metadata management method, which can be applied to distributed storage systems or distributed database systems to achieve dynamic allocation and load balancing of metadata. Figure 1 As shown, the method may include the following steps: In step S110, multi-dimensional attribute information of metadata is obtained.

[0021] In this embodiment, when new metadata needs to be stored or existing metadata needs to be reallocated, the system first obtains the multi-dimensional attribute information of the metadata. This attribute information can be extracted from the metadata itself or indirectly obtained through the data object described by the metadata. The multi-dimensional attribute information aims to characterize the metadata from different perspectives, so as to subsequently generate sharding keys that reflect data distribution requirements. For example, multi-dimensional attribute information may include, but is not limited to, file paths, user identifiers, and access frequencies. The file path reflects the data's storage structure and logical relationships, the user identifier reflects the user dimension to which the data belongs, and the access frequency characterizes the popularity of the metadata. This attribute information can be represented in hash values ​​or other numerical forms for ease of calculation.

[0022] In step S120, a multidimensional sharding key is generated based on the multidimensional attribute information and dynamically adjusted weight coefficients, and the metadata is allocated to the corresponding storage nodes according to the multidimensional sharding key to form metadata shards.

[0023] The weighting coefficient is used to balance the influence of different dimensional attribute information on the generation of shard keys.

[0024] After acquiring multi-dimensional attribute information, this step uses this information and dynamically adjusted weighting coefficients to calculate and generate a multi-dimensional sharding key. This multi-dimensional sharding key is a comprehensive numerical value or identifier used to determine which storage node the metadata should belong to. Specifically, the system assigns a weighting coefficient to each dimension attribute, reflecting the importance of that dimension in the sharding decision. For example, the file path hash, user identifier hash, and access frequency value correspond to weighting coefficients α, β, and γ, respectively. The multi-dimensional sharding key can be calculated using a weighted summation, such as: Multi-dimensional sharding key = α × Hash(file path) + β × Hash(user identifier) ​​+ γ × access frequency value. Of course, other fusion algorithms can also be used.

[0025] It should be noted that the weighting coefficients are dynamically adjusted. This means that the system can adaptively adjust the weights of each dimension based on factors such as real-time operating conditions, business scenarios, or historical data to optimize data distribution. For example, in scenarios where user data access is highly correlated, the weight of the user identifier dimension can be appropriately increased; while in scenarios where file access frequencies vary significantly, the weight of the access frequency dimension can be dynamically adjusted to avoid hotspot concentration.

[0026] After generating a multidimensional sharding key, the system maps metadata to the corresponding storage nodes based on this key. The mapping method can employ conventional routing algorithms such as consistent hashing or range partitioning. Once the allocation is complete, metadata belonging to the same sharding key range naturally forms a metadata shard and is stored on the corresponding node.

[0027] In step S130, the real-time load status of each storage node is obtained.

[0028] To achieve dynamic load balancing, this embodiment requires real-time monitoring of the operating status of each storage node in the system. Real-time load status may include, but is not limited to, metrics such as CPU utilization, memory usage, network bandwidth utilization, and disk I / O. This load information can be collected through a distributed monitoring mechanism, such as each node periodically broadcasting its load information to other nodes, or through a centralized monitoring service. This embodiment preferably employs a decentralized, lightweight monitoring approach, where nodes exchange load information via heartbeat packets, allowing each node to obtain a global or local load view.

[0029] In step S140, when it is determined that load balancing needs to be performed based on the real-time load status, a shard migration operation is triggered to migrate the metadata shards from the source node to the target node.

[0030] Based on the acquired real-time load status, the system can determine whether there is a load imbalance. For example, a threshold can be set: when the overall load of a node exceeds a certain percentage (e.g., 30%) of the cluster's average load, the node is considered overloaded, and some metadata shards need to be migrated to other nodes with lower loads. The judgment logic can be executed autonomously by the overloaded node or by the central scheduler.

[0031] When migration is determined to be necessary, the system triggers a shard migration operation. The migration process typically includes: selecting the metadata shard to migrate (e.g., prioritizing shards with lower access frequency to minimize migration impact), determining the target node (e.g., selecting the node with the lowest load), and then executing the data migration. During the migration process, to ensure uninterrupted service, a temporary redirection strategy can be used to temporarily redirect client requests for the migrated shard to the target node. After the migration is complete, the global metadata routing table is updated to ensure that subsequent accesses can be correctly routed to the new location.

[0032] Through the above steps, this embodiment achieves dynamic allocation of metadata and adaptive load balancing, effectively avoiding hotspot issues caused by static sharding, and improving the system's scalability and responsiveness.

[0033] The metadata management method provided in this application introduces multi-dimensional attribute information and dynamically adjusted weight coefficients to generate sharding keys, which can adaptively adjust the data distribution according to the real-time load status and avoid hotspot concentration. At the same time, combined with distributed load monitoring and automated sharding migration mechanism, the system can dynamically balance the load of each node without interrupting service, which significantly improves the scalability, stability and resource utilization of the distributed storage system.

[0034] Based on the above embodiments, in another embodiment provided in this application, the generation method of the multi-dimensional sharding key is further refined. Specifically, the aforementioned multi-dimensional attribute information may include, but is not limited to, file path hash value, user identifier hash value, and access frequency value. The file path hash value is obtained by hashing the complete path of the file described by the metadata, and is used to reflect the storage structure and logical relationship of the data; the user identifier hash value is obtained by hashing the identifier of the user to whom the file belongs, and is used to aggregate data from the user dimension; the access frequency value is used to characterize the popularity of the metadata access, and can be obtained by counting the number of accesses per unit time.

[0035] Based on the above multi-dimensional attribute information, in order to describe in detail how to generate multi-dimensional sharding keys, step S120 may specifically include the following steps: In step S121, the hash value corresponding to the file path, the hash value corresponding to the user identifier, and the access frequency value of the metadata are obtained.

[0036] When performing metadata allocation, the system first extracts or calculates three key attribute values ​​corresponding to the current metadata. For file paths and user identifiers, the system uses a preset hash algorithm (such as MurmurHash, CRC32, etc.) to convert them into fixed-length hash values. For access frequency values, the system can obtain them from the access statistics of the metadata, such as the number of accesses in the past hour, and normalize them to a numerical range that matches the hash value for subsequent weighted calculation.

[0037] In step S122, based on the dynamically adjusted weighting coefficients, the file path hash value, user identifier hash value, and access frequency value are multiplied by their respective weighting coefficients and then summed in a weighted manner to generate a multidimensional sharding key.

[0038] After obtaining the above three attribute values, the system performs a fusion calculation based on the currently dynamically adjusted weighting coefficients. Assuming the weighting coefficient for the file path hash value is α, the weighting coefficient for the user identifier hash value is β, and the weighting coefficient for the access frequency value is γ, the formula for calculating the multidimensional sharding key can be expressed as: Multidimensional fragment key = α × file path hash value + β × user identifier hash value + γ × access frequency value (1) In this model, α, β, and γ are non-negative real numbers, and their sum can be normalized to 1 or used directly as weighting coefficients without normalization, depending on actual needs. These weighting coefficients are not fixed but dynamically adjusted based on factors such as system operating status and business scenarios, for example, through real-time optimization using reinforcement learning models. This weighted summation method integrates information from multiple dimensions, providing a more comprehensive reflection of metadata distribution requirements.

[0039] It should be noted that the above weighted summation is only an exemplary fusion method. Those skilled in the art can also use other fusion algorithms, such as weighted product, nonlinear combination, etc., as long as they can integrate attribute information from multiple dimensions to generate key values ​​for sharding.

[0040] The method provided in this embodiment generates metadata sharding keys that can distribute data more evenly, while taking into account the logical correlation and access frequency of the data, laying a good foundation for subsequent load balancing.

[0041] This embodiment uses file path hash value, user identifier hash value, and access frequency value as multi-dimensional attribute information, and generates multi-dimensional sharding keys by weighted summation. This enables sharding decisions to simultaneously consider data storage structure, user dimension, and access frequency, achieving fine-grained control of metadata distribution. At the same time, the dynamic adjustment of weight coefficients provides the possibility of adaptive optimization under different business scenarios, further improving the uniformity of data distribution and the overall load balancing of the system.

[0042] In another embodiment provided in this application, the dynamic adjustment method of the aforementioned weight coefficients is specifically defined. To achieve adaptive optimization of the sharding key, the dynamically adjusted weight coefficients can be updated in real time using a reinforcement learning model. Specifically, this process can be implemented through the following steps: (1) The system status is continuously monitored through a reinforcement learning model. The system status includes the real-time load of each storage node and the data access latency.

[0043] As an intelligent decision-making entity, the reinforcement learning model needs to perceive the current operating status of the system to make optimization decisions. In this embodiment, the model's state space consists of multiple key monitoring indicators, which include at least the real-time load of each storage node (such as CPU utilization, memory utilization, network bandwidth utilization, etc.) and data access latency (such as the average response time for metadata queries or read / write operations). This monitoring data can be collected and aggregated in real time by a distributed monitoring system, or reported by each node to a central node for processing. To facilitate model processing, the raw monitoring data can be standardized, and key statistical features (such as mean, variance, maximum value, etc.) can be extracted as state inputs.

[0044] (2) Generate a weight adjustment vector based on the system state. The weight adjustment vector represents the increment of the weight coefficients corresponding to the attribute information of each dimension.

[0045] Upon receiving the current system state, the reinforcement learning model outputs an action based on a pre-defined policy network or value network. In this embodiment, this action is defined as a weight adjustment vector, such as (δw1, δw2, δw3), where each component corresponds to the increment of the weight coefficient of a dimension attribute (such as file path hash, user identifier hash, or access frequency). This increment can be positive, negative, or zero, indicating whether the contribution of that dimension needs to be increased, decreased, or maintained based on the existing weights. In this way, the model can learn the optimal weight configuration direction under the current load mode based on real-time feedback.

[0046] (3) The current weight coefficients are progressively optimized and adjusted according to the weight adjustment vector to obtain the updated weight coefficients.

[0047] After obtaining the weight adjustment vector, the system updates the current weight coefficients. To ensure system stability and avoid drastic fluctuations in data distribution caused by sudden weight changes, this embodiment adopts a gradual adjustment strategy. The update formula can be expressed as: New weight = Old weight + α × Weight adjustment vector (2) Here, α is the step size coefficient, a positive number less than 1, used to control the magnitude of each adjustment. By introducing the step size coefficient, the model only makes small adjustments to the weights each time, allowing the system to converge smoothly to the optimal state. The updated weight coefficients will be used for subsequent multidimensional piecewise key calculations, thus forming a closed-loop optimization process of "monitoring-decision-adjustment".

[0048] It should be noted that reinforcement learning models can be trained using online learning methods, which involves continuously collecting state-action-reward data during system operation to optimize model parameters; or a combination of offline training and online fine-tuning can be used. The reward function can be designed based on comprehensive indicators such as load balancing, average access latency, and node throughput to guide the model towards optimizing for improved overall system performance.

[0049] The method provided in this embodiment eliminates the static configuration or manually set parameters of the weighting coefficients. Instead, they become dynamic variables that can be automatically optimized based on the real-time operating status of the system, enabling the sharding strategy to adaptively respond to load changes.

[0050] This embodiment introduces a reinforcement learning model to dynamically adjust the weight coefficients, thereby achieving adaptive optimization of the sharding strategy. This enables the system to autonomously adjust the importance of each dimension attribute in the sharding decision based on real-time load conditions and access latency, thus maintaining the optimal load balancing effect in dynamically changing business scenarios and improving the system's intelligence and robustness.

[0051] Based on the above embodiments, in another embodiment provided in this application, the specific implementation of step S130 is defined in detail. To achieve decentralized load awareness and autonomous decision-making, this embodiment employs a distributed lightweight monitoring mechanism. Specifically, step S130 may include the following steps: In step S131, each storage node periodically broadcasts its own load information to neighboring nodes. The load information includes CPU utilization, memory utilization, and network bandwidth utilization.

[0052] In this embodiment, each storage node plays a dual role as both a monitor and a monitored node. Each node broadcasts its real-time load information to one or more neighboring nodes with which it has a network connection at preset time intervals (e.g., every 1 second or 5 seconds) via a lightweight communication protocol (such as UDP or a custom heartbeat protocol). This load information includes at least key metrics reflecting the node's workload: CPU utilization (reflecting the use of computing resources), memory utilization (reflecting the use of storage resources), and network bandwidth utilization (reflecting the level of network I / O activity). Of course, in practical applications, other metrics, such as disk I / O latency and disk utilization, can be added according to system requirements. Through this timed broadcasting mechanism, each node can continuously obtain the load status of surrounding nodes without relying on a centralized monitoring service, avoiding single points of failure and performance bottlenecks.

[0053] In step S132, the comprehensive load value is calculated based on the received neighbor node load information and the load information of the node itself.

[0054] After receiving load information broadcast by its neighbors, each node calculates a comprehensive load value by combining it with its own load information. The comprehensive load value is a scalar indicator used to quantify the overall busyness of the nodes; it integrates resource usage from multiple dimensions. The calculation method can be a weighted summation, for example: Total load = a × CPU utilization + b × memory utilization + c × network bandwidth utilization (3) Here, a, b, and c are preset weighting coefficients used to balance the contribution of different resource types to node load. These coefficients can be statically configured or dynamically adjusted according to system characteristics. Through this calculation, each node can quantify its own load level and, based on received neighbor information, calculate the comprehensive load value of neighboring nodes, providing a basis for subsequent comparison and decision-making.

[0055] In step S133, based on the comparison between the overall load value and the average load of the cluster, each storage node independently determines whether to trigger shard migration.

[0056] After obtaining the overall load value, each node needs to determine whether it is currently overloaded, and thus decide whether to initiate a shard migration. This determination is based on comparing its own overall load value with the cluster's average load value. The cluster average load value can be obtained in various ways, such as by calculating the average based on the load information of all neighboring nodes, or by propagating and aggregating it throughout the cluster using distributed algorithms such as the gossip protocol. This embodiment sets a trigger threshold; for example, when a node's own overall load exceeds 1.3 times the cluster average load (i.e., 30%), the node determines itself to be an overloaded node and needs to proactively initiate a shard migration to transfer some of the load to other nodes. This autonomous judgment mechanism allows each node to make independent decisions based on real-time conditions, without waiting for instructions from the central scheduler, thereby greatly improving the system's response speed to load changes.

[0057] Through the above steps, this embodiment realizes a decentralized load monitoring and autonomous decision-making mechanism, in which each node can perceive the local or even global load status in real time and autonomously trigger load balancing operations according to preset rules.

[0058] This embodiment achieves decentralized real-time load monitoring by periodically broadcasting load information and autonomously calculating the comprehensive load value through distributed nodes, avoiding single points of failure and performance bottlenecks caused by centralized monitoring. At the same time, each node autonomously determines whether to trigger shard migration based on the comparison result with the cluster average load, enabling the system to quickly respond to local overload situations and improving the real-time performance of load balancing and the system's self-organizing capabilities.

[0059] Based on the above embodiments, in another embodiment provided in this application, in order to specifically illustrate how to trigger the shard migration operation and migrate the metadata shard from the source node to the target node, the above step S140 may specifically include the following steps: In step S141, if the source node needs to trigger shard migration, the metadata shard to be migrated is determined.

[0060] When a source node determines it is overloaded based on real-time load status (e.g., the overall load exceeds a preset threshold for the cluster's average load), it triggers the shard migration process. The source node first needs to select suitable shards from its managed metadata shards as the objects to be migrated. The selection strategy can be based on various factors, such as the shard's access frequency, shard size, and the shard's correlation with other data. In the preferred approach, to minimize the impact of the migration operation on normal system operation, the source node can prioritize migrating the metadata shards with the lowest access frequency. This is because shards with low access frequency trigger fewer request redirects during migration, reducing the impact on clients. The source node obtains the access frequency value of each shard by statistically analyzing its historical access records, sorts them from low to high frequency, and selects one or more of the top-ranked shards as the shards to be migrated.

[0061] In step S142, based on the migration request received by the target node from the source node, the local metadata routing table is updated after the target node confirms that the resources are available.

[0062] After the source node identifies the shard to be migrated, it needs to select a suitable target node to receive it. The selection of the target node can be based on load monitoring information, such as choosing the neighboring node with the lowest overall load. The source node sends a migration request to the selected target node, which includes at least the identifier of the shard to be migrated, the shard size, and the estimated load. Upon receiving the migration request, the target node first assesses its own resource status to confirm whether it has sufficient storage space, computing power, and network bandwidth to receive the shard. If resources are sufficient, the target node returns an acknowledgment response to the source node and updates its local metadata routing table, adding temporary routing information for the shard to be migrated to correctly handle any redirected requests during the migration process. If resources are insufficient, the target node can reject the request, and the source node will then select another target node.

[0063] In step S143, during the shard migration process, client requests for the shard to be migrated are temporarily redirected to the target node.

[0064] After confirming the migration relationship, the source node begins transmitting the data of the shard to be migrated to the target node. During the data migration process, to ensure the continuity of client services, the system needs to handle access requests during the migration. Specifically, the source node marks the shard as being in a "migration" state in its local routing table or cache. When a client initiates a metadata access request for that shard, the request first reaches the source node (based on the original routing information). The source node recognizes that the shard is being migrated and returns a temporary redirect response (such as an HTTP 307 status code) to the client, which includes the address information of the target node; alternatively, the source node acts as a proxy and forwards the request to the target node. After receiving the redirect information, the client sends the current request to the target node and can cache the target node information. Subsequent related requests can be directly sent to the new node. This temporary redirect mechanism ensures that client requests do not fail during the migration process, achieving seamless migration.

[0065] In step S144, after the migration is completed, the global metadata routing table is updated.

[0066] Once all data for the shard to be migrated has been successfully transmitted to the target node and verified as complete by the target node, the migration operation enters the completion phase. At this point, the system's global metadata routing table needs to be updated to reflect the latest changes in the shard's location. The global metadata routing table can be stored on a separate metadata server or synchronized among nodes via a distributed consensus protocol (such as Raft or Paxos). The update operation changes the owner node of the shard to be migrated from the source node to the target node. After the global routing table is updated, the system notifies all relevant nodes (or invalidates the old cache through the routing table version number mechanism) to ensure that subsequent accesses can be directly routed to the target node. After confirming that the global routing table has been updated and that subsequent requests no longer point to itself, the source node can release the locally stored shard data, completing the entire migration process.

[0067] This embodiment achieves uninterrupted service migration by having the source node autonomously select the shard to be migrated, the target node confirm the resources, request temporary redirection during migration, and update the global routing table after migration. This ensures high availability of the system during load balancing. At the same time, the temporary redirection mechanism ensures the continuity of client requests and data consistency, avoiding access failures or data errors caused by migration, and significantly improving the dynamic adjustment capability and user experience of the distributed storage system.

[0068] Based on the above embodiments, in another embodiment provided in this application, the method for determining the metadata fragments to be migrated in step S141 is further refined, and optimization measures for the migration process are introduced to improve migration efficiency and system stability. Specifically, this embodiment may include the following: 1) Selection strategy for the fragments to be migrated.

[0069] After the source node determines that a shard migration needs to be triggered, it first needs to select suitable migration targets from the multiple metadata shards it manages. To minimize the impact of the migration operation on normal system operation, this embodiment provides an optimization strategy based on access frequency, specifically including: (1) Obtain the access frequency of each metadata shard through the source node.

[0070] The source node maintains access statistics for each metadata shard, such as the number of read and write requests within a past time window (e.g., the last 5 minutes, 1 hour, or 24 hours). These statistics are updated in real time and stored in the node's memory or a local database. When selecting shards to migrate, the source node iterates through all the shards it manages and obtains the access frequency value for each shard. The access frequency can be an absolute number or a normalized relative value, used to characterize the shard's "popularity."

[0071] (2) Prioritize the metadata shard with the lowest access frequency as the shard to be migrated.

[0072] After obtaining the access frequency of each shard, the source node sorts them from lowest to highest access frequency and prioritizes one or more shards with the lowest access frequency as the shards to be migrated. Selecting low-frequency shards for migration has the following advantages: First, low-frequency shards trigger fewer client request redirects during migration, minimizing the impact on user services; second, low-frequency shards have a lower probability of data change, which helps ensure data consistency during migration; finally, migrating low-frequency shards can free up resources on the source node without causing sudden access pressure on the target node. Of course, in practical applications, other factors such as shard size and data correlation can be considered for a comprehensive decision, but access frequency, as the primary consideration, can simply and effectively achieve a smooth migration.

[0073] 2) Optimization measures for the migration process.

[0074] To further reduce the negative impact of the migration process on the system, lower migration costs, and improve efficiency, this embodiment also introduces the following two optional optimization measures, which can be implemented individually or in combination: (1) The shard migration operation adopts the incremental migration method, which only migrates the part of the metadata shard that has changed.

[0075] Traditional full migration requires copying all data from the entire shard from the source node to the target node. When the shard data volume is large, this consumes significant network bandwidth and I / O resources, resulting in a lengthy migration time. This embodiment employs an incremental migration strategy, migrating only the data that has changed since the last synchronization after the initial full synchronization. Specifically, the source node can record a change log for the shard data or use a version number mechanism to continuously track data modifications during the migration process. After completing the basic data synchronization, the source node continuously pushes incremental data (such as newly added, modified, or deleted metadata records) to the target node until the data on both sides is completely consistent. This approach significantly reduces data transfer volume, shortens migration time, and reduces the consumption of system resources.

[0076] (2) Supports parallel migration of multiple metadata shards and uses distributed locks to mark shards during migration.

[0077] When multiple nodes in the system are overloaded or require simultaneous load balancing, serial shard migration may not alleviate the pressure in time. This embodiment supports parallel migration, allowing multiple shards to migrate simultaneously between different node pairs. For example, node A can migrate shard 1 to node B, while node C migrates shard 2 to node D, or the same source node can migrate different shards to multiple target nodes in parallel. Parallel migration can significantly improve the speed of cluster load balancing adjustments.

[0078] However, parallel migration can lead to resource contention or data inconsistency issues. For example, two different migration tasks might attempt to operate on the same shard simultaneously, or a node might fail during the migration process. To address this, this embodiment introduces a distributed lock mechanism to mark shards being migrated. Before starting a migration task, the source node attempts to acquire a distributed lock (e.g., a lock service based on ZooKeeper, etcd, or Redis) for the shard to be migrated. Migration can only begin after successful acquisition, during which other nodes cannot modify or re-migrate the shard. The lock is released after migration is complete. This distributed lock ensures the safety of concurrent migration operations, avoiding conflicts and dirty data.

[0079] It should be noted that the incremental migration and parallel migration described above can be used in combination: when migrating multiple shards in parallel, each shard adopts its own incremental migration method, thereby further improving the migration speed while ensuring efficiency.

[0080] This embodiment prioritizes the metadata shards with the lowest access frequency as migration targets, effectively reducing the impact of the migration process on online services and improving user experience. At the same time, the introduction of an incremental migration strategy significantly reduces data transmission volume, shortens the migration window, and reduces system resource consumption. Parallel migration is supported and coordinated using distributed locks, which improves the speed of load balancing adjustments while ensuring data consistency and operational security, thereby comprehensively optimizing the efficiency and reliability of shard migration.

[0081] As a specific implementation of the above embodiments, in a scenario embodiment provided by this application, the metadata management method provided by this application may specifically include: 1) Dynamic sharding strategy.

[0082] This embodiment introduces more comprehensive multi-dimensional considerations as the sharding key, building upon the traditional hash function. These influencing factors can be dynamically adjusted according to the usage scenario, including but not limited to file path hash values, user ID hash values, and access frequency weights. This composite sharding key design significantly reduces the problem of uneven data distribution, enabling metadata to be distributed more evenly among nodes. The specific design is as follows: (1) File path hash value (Hash(File)).

[0083] It can reflect the storage structure and logical relationship of the data. Taking it into account as a sharding key can make data with similar path structures grouped in close nodes, which is convenient for management and retrieval.

[0084] (2) User ID hash value (Hash(User_Id)).

[0085] It can ensure that the data of the same user is distributed as much as possible on the same or adjacent nodes from the user's perspective, thereby improving the efficiency of user data access.

[0086] (3) Frequency weight.

[0087] This value is assigned based on the frequency of data access. Data that is accessed more frequently will be distributed more evenly across nodes during sharding, in order to avoid hot data being concentrated on a few nodes and affecting the overall system performance.

[0088] Combining the three influencing factors mentioned above, the calculation formula for the multidimensional fragmentation bond can be obtained as follows: Multidimensional sharding key = α × Hash(File) + β × Hash(User_Id) + γ × Frequency.

[0089] Where α, β, and γ are weight coefficients adjusted for practical application scenarios. This embodiment employs a reinforcement learning model to dynamically adjust these weight coefficients. Specifically, an intelligent dynamic adjustment mechanism is designed that continuously monitors several key indicators: ① Real-time load status of each node, including CPU, memory, etc.; ② Data access latency; ③ The weight allocation of the currently used sharding key.

[0090] These monitoring metrics constitute the state space of the reinforcement learning model, which standardizes the original monitoring data and extracts key statistical features.

[0091] Furthermore, the system performs a small adjustment at regular intervals, with the adjustment range kept extremely small to prevent drastic weight changes that could cause model oscillations. The weight adjustment formula is as follows: action = (δ_w1,δ_w2,δ_w3) new_weights = old_weights + α × action Where `action` represents the weight adjustment vector output by reinforcement learning, `α` represents the increment of the weights of the three factors, and `α` is the step size coefficient that controls the adjustment magnitude, used to prevent excessive weight changes from causing system oscillations. In this way, the system can progressively optimize the sharding key weights based on real-time monitoring data, thereby achieving better load balancing and performance response. Where `new_weights` represents the new weights and `old_weights` represents the old weights, see the specific implementation corresponding to formula (2) above.

[0092] Simultaneously, this mechanism analyzes load patterns in real time for different business scenarios, automatically matches pre-set strategy templates, and adjusts the initial weight configuration of the model in a timely manner. For example, in an application scenario primarily focused on user data access, if user data is highly correlated, the value of β can be appropriately increased to concentrate data from the same user; while in a scenario with a complex file storage structure and significant differences in file access frequency, the values ​​of α and γ can be dynamically adjusted according to the specific situation to achieve the best load balancing effect. Multidimensional sharding keys, such as... Figure 2 As shown: This multi-dimensional sharding key design can significantly reduce the problem of uneven data distribution, enabling metadata to be distributed more evenly among nodes, effectively improving the overall performance and stability of the system, and providing solid technical support for various complex distributed application scenarios.

[0093] Meanwhile, to ensure the system maintains a consistently good load balance, this application proposes a shard migration mechanism. The system periodically monitors node load, with monitoring metrics including CPU utilization, memory utilization, and network bandwidth usage. Based on these metrics, shard migration is triggered when the load on a node exceeds a threshold. The migration process is as follows: 1. Select the shard to migrate; 2. Determine the migration node and update the local metadata routing table; 3. During the migration process, client access requests are temporarily redirected to the target node; 4. Wait for the migration to complete and update the global metadata routing table to ensure data consistency across all nodes.

[0094] 2) Distributed load monitoring.

[0095] This embodiment employs a lightweight monitoring protocol to achieve real-time monitoring of the load on each node. Each node periodically broadcasts its own load information to its neighboring nodes. This information, included in heartbeats, covers key metrics such as CPU utilization and memory usage. (See diagram below.) Figure 3 As shown: like Figure 3 The diagram shown is a schematic representation of a distributed load monitoring system provided in an embodiment of this application. It illustrates the specific interaction method in a distributed storage system where storage nodes exchange load information via a lightweight communication protocol.

[0096] exist Figure 3 The diagram shows six storage nodes (nodes 1 to 6), each maintaining its own real-time load status. Taking node 1 as an example, its current load information includes CPU utilization of 80% and memory utilization of 60%; node 2's load information is CPU utilization of 60% and memory utilization of 40%; node 3's is CPU utilization of 70% and memory utilization of 50%; node 4's is CPU utilization of 60% and memory utilization of 40%; node 5's is CPU utilization of 50% and memory utilization of 30%; and node 6's is CPU utilization of 80% and memory utilization of 60%. It should be noted that... Figure 3 The example only shows two metrics: CPU utilization and memory utilization. In practical applications, load information can also include other metrics such as network bandwidth utilization and disk I / O.

[0097] like Figure 3As shown by the middle arrow, each node periodically broadcasts its load information to one or more neighboring nodes with which it has a network connection. For example, node 1 broadcasts its load information to nodes 2 and 3; node 2 broadcasts to nodes 1 and 4; node 3 broadcasts to nodes 1, 4, and 5; node 4 broadcasts to nodes 2, 3, and 6; node 5 broadcasts to nodes 3 and 6; and node 6 broadcasts to nodes 4 and 5. Through this periodic broadcasting mechanism, each node can continuously obtain the real-time load status of its surrounding neighboring nodes, thus forming a decentralized load information dissemination network.

[0098] Based on the received load information from neighboring nodes and its own load information, each node can calculate its overall load value and compare it with the cluster average load. For example, node 1, based on the load information received from nodes 2 and 3 and its own load, can calculate the average load within a local area and determine whether it is overloaded (e.g., its overall load exceeds 30% of the average load). When a node determines that it is overloaded, it can autonomously trigger a shard migration operation to migrate some metadata shards to neighboring nodes with lower loads.

[0099] Figure 3 The distributed load monitoring mechanism shown avoids single points of failure and performance bottlenecks caused by centralized monitoring nodes. Each node achieves global or local load awareness by exchanging load information, providing a data foundation for subsequent autonomous load balancing decisions. This mechanism, in conjunction with steps S130 to S140 described in the embodiments of this application, jointly realizes dynamic allocation and adaptive scheduling of metadata.

[0100] The formula for calculating the overall load is as follows: Total load = α × CPU utilization + β × memory utilization + γ × network bandwidth utilization.

[0101] In this way, α, β, and γ enable each node to promptly understand the load status of its surrounding nodes.

[0102] Based on the load information obtained from the aforementioned lightweight monitoring protocol, each node can autonomously decide whether to migrate shards according to its own load and the information of its neighboring nodes. The conditions for triggering migration are as follows: The total load of a node is greater than the average total load of the cluster by 1.3.

[0103] This autonomous decision-making approach fully leverages the autonomy of each node, enabling the system to make timely adjustments based on actual load conditions, effectively improving the system's response speed and stability.

[0104] 3) Automated fragment scheduling.

[0105] This embodiment aims to design an automated sharding scheduling system that can automatically determine how to perform shard migration based on the current system state, such as... Figure 4 As shown.

[0106] like Figure 4 The diagram shown is a shard migration illustration provided in an embodiment of this application. It illustrates the specific process and inter-node interactions in a distributed storage system when a source node determines it is overloaded, triggering a shard migration operation.

[0107] exist Figure 4 The diagram shows four storage nodes (node ​​1, node 3, node 4, and node 5), each currently managing several metadata shards, and labeling each shard with its access frequency level (high, medium, and low) to characterize the shard's popularity.

[0108] Specifically, node 1 is designated as the overloaded node (source node), and its currently managed metadata shards include: shard 1 (high access frequency), shard 2 (medium access frequency), and shard 3 (low access frequency). According to the migration strategy provided in this embodiment, the source node preferentially selects the shard with the lowest access frequency as the target shard to be migrated, in order to minimize the impact of the migration operation on the system. Therefore, node 1 marks shard 3 as the shard to be migrated and initiates a migration request to the target node.

[0109] As shown by the arrow in the diagram, Node 1 sends a migration request to Node 3. Node 3, as a potential target node, needs to confirm whether its resources are sufficient to receive the shard after receiving the migration request. Although the specific load information of Node 3 is not shown in the diagram, it can be inferred from the context that Node 3 should be one of the selected nodes with a low load.

[0110] The diagram also shows the shard distribution of other nodes: node 4 manages shard 1 (medium access frequency); node 5 manages both shard 1 (low access frequency) and shard 2 (medium access frequency). These nodes demonstrate the diversity of shard distribution in the system and provide candidates for selecting the target node for migration.

[0111] During the migration process, Node 1 transmits the data of shard 3 to Node 3. During this time, if a client requests access to shard 3, the request will first reach Node 1 (based on the original routing information). Once Node 1 recognizes that the shard is being migrated, it will temporarily redirect the request to Node 3 to ensure uninterrupted service. After the migration is complete, the system updates the global metadata routing table, changing the home node of shard 3 from Node 1 to Node 3, and subsequent access will be directly routed to Node 3.

[0112] Figure 4The shard migration process shown is in conjunction with steps S141 to S144 described in the embodiments of this application, intuitively demonstrating the complete process from selecting the shard to be migrated, sending the migration request, data migration to the final completion of the migration, and embodying the core idea of ​​the automated shard scheduling mechanism of this application.

[0113] Specifically: (1) Select the shard to be migrated from the source node.

[0114] The source node considers multiple factors when selecting shards to migrate, and in this embodiment, access frequency is taken as an important metric. Typically, the source node prioritizes migrating shards with the lowest access frequency to minimize the impact of the migration operation on normal system operation. For example, in a storage system containing a large amount of historical data, shards containing historical data that are rarely accessed will be prioritized for migration.

[0115] (2) The target node confirms receipt of the fragment and updates the metadata routing table.

[0116] Upon receiving a migration request from the source node, the target node first checks if it has sufficient resources to receive the shard. If it confirms that it can receive it, the target node updates its metadata routing table to ensure that subsequent requests for that shard can be processed correctly.

[0117] (3) During the migration process, the client requests are temporarily redirected.

[0118] During the migration process, client requests are temporarily redirected to the target node: To ensure that client requests are processed correctly during shard migration, the system temporarily redirects client requests for the shards to be migrated to the target node and caches the target node information. Subsequent requests are then sent directly to the new node. This ensures data continuity and consistency, preventing client request failures or data retrieval errors.

[0119] (4) After the migration is completed, update the global metadata routing table.

[0120] Once the shard migration is complete, the system will update the global metadata routing table so that the entire system can obtain the latest location of the shard, thereby ensuring that all subsequent nodes can correctly access and operate on the data.

[0121] Meanwhile, to further reduce the negative impact of the migration process on the system and reduce migration costs, this application's embodiments employ incremental migration and concurrent migration for optimization.

[0122] In this embodiment, incremental migration can be performed. That is, only a portion of the data in the hot shard is migrated, rather than the entire shard. For example, in a frequently updated database, for a certain hot shard, only the data that has recently changed is migrated, which can greatly reduce the amount of data transferred and shorten the migration time.

[0123] In this embodiment, parallel migration is also possible. This means that multiple shards can be migrated simultaneously, significantly improving migration efficiency through parallel processing. In a large-scale distributed storage system, when multiple nodes experience loads exceeding thresholds and require shard migration, the system can initiate multiple migration tasks concurrently, allowing different shards to migrate between different node pairs, thereby accelerating the load balancing adjustment of the entire system. Distributed locks are used to mark shards during migration to prevent conflicts arising from concurrent migrations.

[0124] In summary, compared with related technologies, the technical solution provided in this application has the following advantages: First, this application's embodiment employs a dynamic sharding strategy. By introducing multi-dimensional attribute information and dynamically adjusted weight coefficients to generate multi-dimensional sharding keys, it can adaptively optimize weight configuration based on application scenarios, cluster historical information, and real-time load status, achieving fine-grained distribution of metadata. This strategy effectively avoids the performance bottleneck caused by static sharding, where some nodes bear too much data or requests. Simultaneously, it supports flexible adjustment of the number and size of shards according to business needs, significantly improving the system's elasticity and scalability.

[0125] Second, the embodiments of this application employ a distributed load monitoring mechanism, distributing monitoring tasks across various storage nodes. Each node periodically exchanges load information via a lightweight protocol, forming a decentralized load-aware network. This design allows the system to easily adapt to scaling, avoiding single points of failure and performance bottlenecks associated with centralized monitoring. Even if some nodes fail, the remaining nodes can still function normally and maintain load monitoring capabilities, thereby significantly improving the system's fault tolerance and availability.

[0126] Third, the embodiments of this application implement an automated sharding scheduling system that can autonomously trigger shard migration operations based on real-time monitoring of node load and dynamically adjust the distribution of metadata among nodes. Through optimization measures such as prioritizing the migration of low-frequency access shards, using incremental migration to reduce data transmission volume, supporting parallel migration to improve efficiency, and using distributed locks to ensure consistency, the system can quickly respond to load changes while ensuring uninterrupted service, keeping the workload of each node relatively balanced, thereby comprehensively improving the overall performance and resource utilization of the distributed storage system.

[0127] By dividing each functional module according to its corresponding function, this application provides a metadata management device, which can be a server, a terminal, or a chip applied to a server. Figure 5 A schematic block diagram of the functional modules of a metadata management device provided for an exemplary embodiment of this application. For example... Figure 5 As shown, the metadata management device includes: Information acquisition module 51 is used to acquire multi-dimensional attribute information of metadata; The multidimensional sharding key generation and allocation module 52 is used to generate multidimensional sharding keys based on the multidimensional attribute information and dynamically adjusted weight coefficients, and allocate the metadata to the corresponding storage nodes according to the multidimensional sharding keys to form metadata shards; wherein, the weight coefficients are used to balance the influence of different dimensional attribute information in the generation of sharding keys; The load status acquisition module 53 is used to acquire the real-time load status of each storage node; The metadata shard migration module 54 is used to trigger a shard migration operation when it is determined that load balancing needs to be performed based on the real-time load status, and to migrate the metadata shard from the source node to the target node.

[0128] In another embodiment provided in this application, the multi-dimensional attribute information includes a file path hash value, a user identifier hash value, and an access frequency value; the multi-dimensional shard key generation and allocation module 52 is specifically used for: Obtain the hash value corresponding to the file path, the hash value corresponding to the user identifier, and the access frequency value of the metadata; Based on dynamically adjusted weighting coefficients, the file path hash value, user identifier hash value, and access frequency value are multiplied by their respective weighting coefficients and then summed in a weighted manner to generate a multidimensional sharding key.

[0129] In another embodiment provided in this application, the multidimensional fragmentation key generation and allocation module 52 is specifically used for: The system status is continuously monitored through a reinforcement learning model, including the real-time load and data access latency of each storage node. A weight adjustment vector is generated based on the system state, and the weight adjustment vector represents the increment of the weight coefficient corresponding to each dimension attribute information; The current weight coefficients are progressively optimized and adjusted based on the weight adjustment vector to obtain the updated weight coefficients.

[0130] In another embodiment provided in this application, the load status acquisition module 53 is specifically used for: Each storage node periodically broadcasts its own load information to neighboring nodes, including CPU utilization, memory utilization, and network bandwidth utilization. Calculate the overall load value based on the received load information of neighboring nodes and its own load information; Based on the comparison between the overall load value and the average load of the cluster, each storage node independently determines whether to trigger shard migration.

[0131] In another embodiment provided in this application, the metadata sharding migration module 54 is specifically used for: If the source node needs to trigger shard migration, determine the metadata shards to be migrated; Based on the migration request received by the target node from the source node, the target node updates its local metadata routing table after confirming that the resources are available. During the shard migration process, client requests for the shards to be migrated will be temporarily redirected to the target node; After the migration is complete, update the global metadata routing table.

[0132] In another embodiment provided in this application, the metadata sharding migration module 54 is further configured to: The access frequency of each metadata shard is obtained through the source node; Prioritize selecting the metadata shard with the lowest access frequency as the shard to be migrated; The shard migration operation adopts an incremental migration method, migrating only the changed data in the metadata shard; It supports parallel migration of multiple metadata shards and uses distributed locks to mark shards during migration.

[0133] The metadata management device provided in this application generates sharding keys by introducing multi-dimensional attribute information and dynamically adjusted weight coefficients. It can adaptively adjust the data distribution according to the real-time load status to avoid hotspot concentration. At the same time, combined with distributed load monitoring and automated sharding migration mechanism, the system can dynamically balance the load of each node without interrupting service, which significantly improves the scalability, stability and resource utilization of the distributed storage system.

[0134] This application also provides an electronic device, including: at least one processor; a memory for storing executable instructions of the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the method disclosed in the embodiments of this application.

[0135] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 6As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can perform the corresponding steps in the methods disclosed in the embodiments of this application.

[0136] The processor 1801 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this application can be implemented by the integrated logic circuitry in the hardware of the processor 1801 or by instructions in software form. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1802, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1801 reads information from the memory 1802 and, in conjunction with its hardware, completes the steps of the above method.

[0137] Furthermore, the various operations / processes according to this application, when implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 7 The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 7 A structural block diagram of a computer system provided for an exemplary embodiment of this application.

[0138] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0139] like Figure 7 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0140] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0141] The computing unit 1901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in the embodiments of this application can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in the embodiments of this application by any other suitable means (e.g., by means of firmware).

[0142] This application also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this application.

[0143] The computer-readable storage medium in this application embodiment may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0144] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0145] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this application.

[0146] In embodiments of this application, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The modules, components, or units described in the embodiments of this application can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0149] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0150] The above description is merely an embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0151] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A metadata management method, characterized in that, The method includes: Retrieve multi-dimensional attribute information of metadata; Based on the multi-dimensional attribute information and dynamically adjusted weight coefficients, a multi-dimensional sharding key is generated, and the metadata is allocated to the corresponding storage nodes according to the multi-dimensional sharding key to form metadata shards; wherein, the weight coefficients are used to balance the influence of different dimensional attribute information in the generation of sharding keys; Obtain the real-time load status of each storage node; When it is determined that load balancing needs to be performed based on the real-time load status, a shard migration operation is triggered to migrate the metadata shard from the source node to the target node.

2. The method according to claim 1, characterized in that, The multi-dimensional attribute information includes file path hash value, user identifier hash value, and access frequency value; The step of generating a multidimensional sharding key based on the multidimensional attribute information and dynamically adjusted weight coefficients includes: Obtain the hash value corresponding to the file path, the hash value corresponding to the user identifier, and the access frequency value of the metadata; Based on dynamically adjusted weighting coefficients, the file path hash value, user identifier hash value, and access frequency value are multiplied by their respective weighting coefficients and then summed in a weighted manner to generate a multidimensional sharding key.

3. The method according to claim 2, characterized in that, The dynamically adjusted weighting coefficients include: The system status is continuously monitored through a reinforcement learning model, including the real-time load and data access latency of each storage node. A weight adjustment vector is generated based on the system state, and the weight adjustment vector represents the increment of the weight coefficient corresponding to each dimension attribute information; The current weight coefficients are progressively optimized and adjusted based on the weight adjustment vector to obtain the updated weight coefficients.

4. The method according to claim 1, characterized in that, The process of obtaining the real-time load status of each storage node includes: Each storage node periodically broadcasts its own load information to neighboring nodes, including CPU utilization, memory utilization, and network bandwidth utilization. Calculate the overall load value based on the received load information of neighboring nodes and its own load information; Based on the comparison between the overall load value and the average load of the cluster, each storage node independently determines whether to trigger shard migration.

5. The method according to claim 4, characterized in that, The triggered shard migration operation, which migrates the metadata shard from the source node to the target node, includes: If the source node needs to trigger shard migration, determine the metadata shards to be migrated; Based on the migration request received by the target node from the source node, the target node updates its local metadata routing table after confirming that the resources are available. During the shard migration process, client requests for the shards to be migrated will be temporarily redirected to the target node; After the migration is complete, update the global metadata routing table.

6. The method according to claim 5, characterized in that, The process of determining the metadata fragments to be migrated includes: The access frequency of each metadata shard is obtained through the source node; Prioritize selecting the metadata shard with the lowest access frequency as the shard to be migrated; And / or, The shard migration operation adopts an incremental migration method, migrating only the changed data in the metadata shard; And / or, It supports parallel migration of multiple metadata shards and uses distributed locks to mark shards during migration.

7. A metadata management device, characterized in that, The device includes: The information acquisition module is used to acquire multi-dimensional attribute information of metadata; The multidimensional sharding key generation and allocation module is used to generate multidimensional sharding keys based on the multidimensional attribute information and dynamically adjusted weight coefficients, and to allocate the metadata to the corresponding storage nodes according to the multidimensional sharding keys to form metadata shards; wherein, the weight coefficients are used to balance the influence of different dimensional attribute information in the generation of sharding keys; The load status acquisition module is used to acquire the real-time load status of each storage node; The metadata shard migration module is used to trigger a shard migration operation when it is determined that load balancing needs to be performed based on the real-time load status, and to migrate the metadata shards from the source node to the target node.

8. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images