Metadata storage method and device, electronic equipment and storage medium

Abstract

The invention provides a metadata storage method and device, electronic equipment and a storage medium, and relates to the technical field of data storage. The method comprises the following steps: acquiring metadata of a file system to be stored, and determining an existing first storage cluster for storing the metadata; then, the total data volume of the metadata is determined, and if the total data volume of the metadata is larger than a preset threshold value, a new second storage cluster used for storing the metadata is created. And next, dynamically setting a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule, and sending the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient. According to the method, the storage efficiency of the metadata and the resource utilization rate of the storage cluster are improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a metadata storage method, device, electronic device and storage medium. Background Art

[0002] Metadata is the core part of the operation of the file system, providing key information about files and directories without including the actual content of the files. In order to ensure the integrity, security and efficiency of the file system and to provide the functions and performance expected by users, the metadata of the file system needs to be stored.

[0003] Metadata is generally stored and managed centrally by multiple storage service nodes in a dedicated storage cluster. Since the storage volume of metadata is usually huge, the storage cluster needs to be expanded. Currently, there are two ways to expand the storage cluster: one is to improve the storage capacity of the storage service nodes by improving the resource configuration of the storage service nodes in the storage cluster; the other is to improve the storage capacity of the storage cluster by adding new storage service nodes in the storage cluster.

[0004] However, the prior art has technical problems of low metadata storage efficiency and low resource utilization of the storage cluster. Summary of the invention

[0005] The present application provides a metadata storage method, device, electronic device and storage medium to solve the technical problems of low metadata storage efficiency and low resource utilization of storage clusters in the prior art.

[0006] In a first aspect, the present application provides a metadata storage method, comprising:

[0007] Obtaining metadata of a file system waiting to be stored, and determining a first storage cluster; wherein the first storage cluster represents an existing cluster for storing metadata;

[0008] Determining a total data volume of the metadata, and if the total data volume is greater than a preset threshold, creating a new second storage cluster for storing the metadata;

[0009] Dynamically setting a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule; and sending the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient.

[0010] Optionally, the first storage cluster includes at least one first storage service, and the second storage cluster includes at least one second storage service;

[0011] If the total data volume is greater than a preset threshold, creating a new second storage cluster for storing metadata includes:

[0012] Determine the total data storage capacity of the first storage service;

[0013] If the total data volume is greater than the total data storage volume, the second storage service is registered, and the second storage cluster is formed based on the registered second storage service.

[0014] Optionally, dynamically setting a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule includes:

[0015] Calculating in real time a first storable data volume of a first storage service included in the first storage cluster, and a second storable data volume of a second storage service included in the second storage cluster;

[0016] Based on the first storable data volume and the second storable data volume, a weight coefficient is dynamically set for the first storage cluster and the second storage cluster.

[0017] Optionally, the method further includes:

[0018] Monitoring the first storage cluster and the second storage cluster;

[0019] When detecting that a first storage service in the first storage cluster and / or a second storage service in the second storage cluster are online, creating a cluster manager for the first storage cluster and / or the second storage cluster;

[0020] When it is detected that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are both in an offline state, the cluster manager of the first storage cluster and / or the second storage cluster is cleared.

[0021] Optionally, the monitoring the first storage cluster and the second storage cluster further includes:

[0022] Sending a heartbeat detection signal to a first storage service in the first storage cluster and a second storage service in the second storage cluster;

[0023] If a heartbeat response is received from the first storage service and / or the second storage service based on the heartbeat detection signal, it is determined that the first storage service and / or the second storage service is in an online state; if the heartbeat response is not received, it is determined that the first storage service and / or the second storage service is in an offline state.

[0024] Optionally, sending the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient includes:

[0025] If a target storage cluster is specified in advance for the metadata, the metadata is sent to the target storage cluster for storage processing; wherein the target storage cluster is the first storage cluster or the second storage cluster;

[0026] If a target storage cluster is not specified in advance for the metadata, a target storage cluster is determined from the first storage cluster or the second storage cluster based on the weight coefficient; and the metadata is sent to the target storage cluster for storage processing.

[0027] Optionally, the method further includes:

[0028] If the weight coefficient of the first storage cluster and / or the second storage cluster is zero, it is determined that the first storage cluster and / or the second storage cluster is in a data saturation state, and metadata is no longer sent to the first storage cluster and / or the second storage cluster.

[0029] In a second aspect, the present application provides a metadata storage device, comprising:

[0030] An acquisition module, used to acquire metadata of a file system waiting to be stored;

[0031] A determination module, configured to determine a first storage cluster and a total amount of data of the metadata; wherein the first storage cluster represents an existing cluster for storing metadata;

[0032] A creation module, configured to create a new second storage cluster for storing metadata if the total data volume is greater than a preset threshold;

[0033] The storage module is used to dynamically set a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule; and send the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient.

[0034] Optionally, the first storage cluster includes at least one first storage service, and the second storage cluster includes at least one second storage service;

[0035] The determination module is further used to determine the total data storage capacity of the first storage service;

[0036] The creation module further includes: a registration module, which is used to register the second storage service if the total data volume is greater than the total data storage volume, and form the second storage cluster based on the registered second storage service.

[0037] Optionally, the storage module further includes: a calculation module and a setting module.

[0038] The calculation module is used to calculate in real time a first storable data volume of a first storage service included in the first storage cluster, and a second storable data volume of a second storage service included in the second storage cluster;

[0039] The setting module is used to dynamically set a weight coefficient for the first storage cluster and the second storage cluster based on the first storable data volume and the second storable data volume.

[0040] Optionally, the metadata storage device further includes: a monitoring module, configured to monitor the first storage cluster and the second storage cluster;

[0041] The creation module is further configured to create a cluster manager for the first storage cluster and / or the second storage cluster when detecting that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are online;

[0042] The metadata storage device also includes: a clearing module, which is used to clear the cluster manager of the first storage cluster and / or the second storage cluster when it is detected that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are both offline.

[0043] Optionally, the monitoring module further includes: a sending module, configured to send a heartbeat detection signal to a first storage service in the first storage cluster and a second storage service in the second storage cluster;

[0044] The determination module is also used to determine that the first storage service and / or the second storage service is in an online state if a heartbeat response is received from the first storage service and / or the second storage service based on the heartbeat detection signal; if the heartbeat response is not received, determine that the first storage service and / or the second storage service is in an offline state.

[0045] Optionally, the storage module is further configured to send the metadata to the target storage cluster for storage processing if a target storage cluster is specified in advance for the metadata; wherein the target storage cluster is the first storage cluster or the second storage cluster;

[0046] The determination module is further configured to determine a target storage cluster from the first storage cluster or the second storage cluster based on the weight coefficient if no target storage cluster is specified in advance for the metadata;

[0047] The storage module is further used to send the metadata to the target storage cluster for storage processing.

[0048] Optionally, the determination module is also used to determine that the first storage cluster and / or the second storage cluster is in a data saturation state if the weight coefficient of the first storage cluster and / or the second storage cluster is zero, and no longer send metadata to the first storage cluster and / or the second storage cluster.

[0049] In a third aspect, the present application provides an electronic device, including:

[0050] Memory and processor;

[0051] Wherein, the memory stores computer-executable instructions;

[0052] The processor executes the computer-executable instructions stored in the memory to implement the metadata storage method as described in the first aspect and various possible implementations.

[0053] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the metadata storage method as described in the first aspect and various possible implementation methods.

[0054] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the metadata storage method as described in the first aspect and various possible implementation methods.

[0055] The metadata storage method, device, electronic device and storage medium provided by the present application obtain the metadata of the file system waiting to be stored, and determine the existing first storage cluster for storing metadata. After that, the total data volume of the metadata is determined. If the total data volume of the metadata is greater than the preset threshold, a new second storage cluster for storing metadata is created. In the process of obtaining the second storage cluster because the total data volume of the metadata is too large and the storage cluster needs to be horizontally expanded, the service of the first storage cluster will not be interrupted, and the metadata can still be stored and processed, thereby ensuring the storage efficiency of the metadata. Next, based on the preset rules, the weight coefficient is dynamically set for the first storage cluster and the second storage cluster, and according to the weight coefficient, the metadata is sent to the first storage cluster or the second storage cluster for storage processing. By dynamically setting the weight coefficient for the first storage cluster and the second storage cluster to apportion the metadata to be stored, it is avoided that some storage clusters are idle and the load of other storage clusters is too high, load balancing between storage clusters is achieved, and the resource utilization of the storage cluster is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] Figure 1 Schematic diagram of the metadata storage method provided in the embodiment of the present application Figure 1 ;

[0058] Figure 2 Application scenarios of the metadata storage method provided in the embodiments of the present application Figure 1 ;

[0059] Figure 3 Application scenarios of the metadata storage method provided in the embodiments of the present application Figure 2 ;

[0060] Figure 4 Schematic diagram of the metadata storage method provided in the embodiment of the present application Figure 2 ;

[0061] Figure 5 A schematic diagram of the structure of a metadata storage device provided in an embodiment of the present application;

[0062] Figure 6 This is a hardware structure diagram of the electronic device provided for this application.

[0063] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0066] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0068] Metadata is data about data. In a file system, metadata only describes the attributes and structure of a file, but does not involve the actual data content of the file. For example, metadata usually includes the file's name, size, type, creation and modification time, permission information, owner information, etc.

[0069] Metadata is a key component of the normal operation of the file system. Without metadata, the file system will not be able to identify, access and manage these files and directories. Therefore, by storing and managing metadata, the file system can ensure its integrity, security and efficiency, and meet users' expectations of the functions and performance of the file system.

[0070] It is understandable that metadata is usually not stored in a decentralized manner, but is centrally stored and managed in a dedicated storage cluster, where a storage cluster is a cluster composed of multiple storage service nodes, used to provide high availability and high performance data storage services.

[0071] Since metadata involves a lot of information about files and directories, especially in large-scale file systems, the amount of metadata storage may be very large, which requires the storage cluster to have sufficient capacity and performance to handle it. Therefore, in order to cope with the growth of metadata storage demand, the storage cluster needs to be expanded. Currently, there are two main ways to expand the storage cluster.

[0072] The first is to improve the storage capacity of storage service nodes by improving the resource configuration of existing storage service nodes in the storage cluster, such as CPU, memory, storage space, etc. This approach may involve hardware upgrades or optimized configurations. The second is to increase the overall storage capacity of the storage cluster by adding more storage service nodes to the storage cluster, that is, increasing the number of nodes to improve capacity and performance.

[0073] However, when upgrading the resource configuration of storage service nodes, the storage cluster needs to be deactivated to replace hardware or update system configuration; when adding new storage service nodes, the network and data distribution need to be reconfigured, which will also cause temporary service interruption. Therefore, during the expansion of the storage cluster, due to the deactivation of some or all functions of the storage cluster, metadata may not be stored normally, thus affecting the storage efficiency of metadata.

[0074] In addition, the storage cluster may also have the problem of unbalanced load distribution, that is, some storage service nodes are overloaded, while other storage service nodes are idle or not effectively used. This situation will lead to resource waste and reduce the overall resource utilization of the storage cluster.

[0075] In response to the above technical problems, when there are too many metadata waiting to be stored, the inventor does not expand on the basis of the original storage cluster, but creates a new storage cluster to achieve the purpose of horizontal expansion. Creating a new storage cluster will not affect the existing storage cluster, which means that when creating a new storage cluster, the existing storage cluster will not be unable to store and process the metadata due to being deactivated, thereby improving the storage efficiency of the metadata. In addition, the inventor also thought of dynamically setting weight coefficients for the existing storage cluster and the newly created storage cluster, and allocating the metadata to the corresponding storage cluster for storage processing according to the weight coefficient, so as to achieve load balancing between storage clusters and improve the resource utilization of the storage cluster.

[0076] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0077] An embodiment of the present application provides a metadata storage method. Figure 1 Schematic diagram of the metadata storage method provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the metadata storage method includes:

[0078] S101: Obtain metadata of a file system waiting to be stored, and determine a first storage cluster.

[0079] The first storage cluster is an existing cluster for storing metadata, that is, a cluster that has been configured and is in use. The metadata of the file system may include information such as the name, size, type, permission, creation and modification time of the file, which is not specifically limited here.

[0080] S102: Determine the total data volume of the metadata, and if the total data volume is greater than a preset threshold, create a new second storage cluster for storing the metadata.

[0081] It should be noted that the first storage cluster includes at least one first storage service.

[0082] It is understandable that the preset threshold should be the maximum amount of data that the existing first storage cluster can store metadata in. In the storage cluster, the storage service is the core component for processing metadata storage and management, so the preset threshold can be set to the total data storage capacity of the first storage service included in the first storage cluster.

[0083] Specifically, after determining the total data storage capacity of the first storage service, the total data capacity of the metadata is compared with the total data storage capacity of the first storage service. If the total data capacity is less than or equal to the total data storage capacity, it indicates that the first storage cluster can store all the metadata; if the total data capacity is greater than the total data storage capacity, it indicates that the first storage cluster cannot store all the metadata, and a new second storage cluster for storing metadata needs to be created.

[0084] Similarly, the second storage cluster also includes at least one second storage service. Since the first storage cluster is a cluster that has been configured and is in use, the first storage service in the first storage cluster has been registered and configured. The second storage cluster is a newly created cluster, so it is necessary to first register the second storage service, and form a second storage cluster based on the registered second storage service. The service registration process can ensure that the second storage service communicates and collaborates correctly with other parts of the second storage cluster.

[0085] It is understandable that, whether for the existing first storage cluster or the newly created second storage cluster, all storage services and resources in the cluster need to be managed in a centralized manner. Therefore, the first storage cluster and the second storage cluster can be monitored. When it is detected that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are online, it means that the first (second) storage cluster is operating normally and can store and process metadata through the online first (second) storage service. At this time, it is necessary to create a cluster manager for the first storage cluster and / or the second storage cluster.

[0086] It should be noted that the cluster manager can automatically discover, register and configure newly added storage services, and manage the life cycle of existing storage services, including the addition, removal, upgrade and fault handling of storage services.

[0087] When it is detected that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are both offline, it means that the first (second) storage cluster cannot provide storage services. At this time, since there is no storage service to be managed, there is no need for the cluster manager to continue running, and the cluster manager of the first storage cluster and / or the second storage cluster can be cleared to release cluster resources.

[0088] However, the first storage cluster and / or the second storage cluster still needs to be monitored. Once it is found that the first storage service and / or the second storage service are online in the first storage cluster and / or the second storage cluster, a new cluster processor will be created for the first storage cluster and / or the second storage cluster.

[0089] It should be noted that a heartbeat mechanism is usually used to monitor the first storage cluster and the second storage cluster, and the heartbeat mechanism confirms the online status and health status of the node by periodically sending a heartbeat detection signal. Specifically, a heartbeat detection signal is sent to the first storage service in the first storage cluster and the second storage service in the second storage cluster. If a heartbeat response is received from the first storage service and / or the second storage service based on the heartbeat detection signal, it is determined that the first storage service and / or the second storage service is in an online state; if no heartbeat response is received, it is determined that the first storage service and / or the second storage service is in an offline state.

[0090] S103: Dynamically set a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule; and send the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient.

[0091] The weight coefficient is a value used to indicate the relative importance or priority of a storage cluster. By dynamically setting the weight coefficient, storage cluster resources can be used more effectively to avoid overloading or wasting resources of certain storage clusters.

[0092] Specifically, this step calculates in real time the first storable data volume of the first storage service included in the first storage cluster and the second storable data volume of the second storage service included in the second storage cluster. Based on the first storable data volume and the second storable data volume, a weight coefficient is dynamically set for the first storage cluster and the second storage cluster.

[0093] For example, if the second storable data volume corresponding to the second storage cluster is calculated to be larger, it means that the second storage cluster can store more metadata, and the weight coefficient of the second storage cluster can be set to be larger; if the first storable data volume corresponding to the first storage cluster is calculated to be smaller, it means that the first storage cluster can store less metadata, and the weight coefficient of the first storage cluster can be set to be smaller.

[0094] It is worth noting that if the weight coefficient of the first storage cluster and / or the second storage cluster is zero, it is determined that the first storage cluster and / or the second storage cluster is in a data saturation state and can no longer store metadata, and metadata is no longer sent to the first storage cluster and / or the second storage cluster.

[0095] It should be explained that the first (second) storable data volume refers to the amount of data that can be stored without exceeding the total storage data volume of all first (second) storage services included in the first (second) storage cluster. Real-time calculation of the first (second) storable data volume of the first (second) storage cluster means dynamically evaluating the currently available storage capacity of each first (second) storage service in the first (second) storage cluster in order to reflect the latest status of the storage resources.

[0096] It should be noted that if a target storage cluster has been specified in advance for the metadata, the metadata is directly sent to the target storage cluster for storage processing, wherein the target storage cluster is the first storage cluster or the second storage cluster. If a target storage cluster has not been specified in advance for the metadata, the target storage cluster is determined from the first storage cluster or the second storage cluster according to the weight coefficient, and the metadata is sent to the target storage cluster for storage processing.

[0097] The metadata can be stored in a hard disk, memory, distributed storage system, etc. through the storage service in the target storage cluster. The specific storage location of the metadata can be determined by the actual application scenario and requirements and is not limited here.

[0098] The metadata storage method provided by the present application obtains metadata of a file system waiting to be stored, and determines an existing first storage cluster for storing metadata. Afterwards, the total data volume of the metadata and the total data storage volume of the first storage service included in the first storage cluster are determined. If the total data volume is greater than the total data storage volume, the second storage service included in the second storage cluster is registered, and a second storage cluster is formed based on the registered second storage service. In the process of obtaining the second storage cluster because the total data volume of the metadata is too large and the storage cluster needs to be horizontally expanded, the service of the first storage cluster will not be interrupted, and the metadata can still be stored and processed, thereby ensuring the storage efficiency of the metadata. Next, the first storable data volume of the first storage service and the second storable data volume of the second storage service are calculated in real time. Based on the first storable data volume and the second storable data volume, a weight coefficient is dynamically set for the first storage cluster and the second storage cluster, and according to the weight coefficient, the metadata is sent to the first storage cluster or the second storage cluster for storage processing. By dynamically setting a weight coefficient for the first storage cluster and the second storage cluster to apportion the metadata to be stored, it is avoided that some storage clusters are idle and the load of other storage clusters is too high, load balancing between storage clusters is achieved, and the resource utilization of the storage clusters is improved.

[0099] Next, through a specific example Figure 1 The metadata storage method shown is further explained. Figure 2 Application scenarios of the metadata storage method provided in the embodiments of the present application Figure 1 ,like Figure 2 As shown, the implementation of the metadata storage method depends on the file system client, the request distributor, the service registration center, and the storage cluster.

[0100] Specifically, the user uploads the metadata of the file system to the request distributor through the file system client. The request distributor is mainly responsible for the scheduling of metadata, which includes a cluster selector and a cluster weight manager. In addition, the request distributor also associates a cluster manager for each storage cluster.

[0101] The cluster manager maintains the storage services of the storage cluster based on the service discovery mechanism, and the request distributor dynamically monitors all storage services in the storage cluster. When all storage services in a storage cluster are offline, the cluster manager associated with the storage cluster will be dynamically cleared, and when the storage service of the storage cluster is online, a new cluster processor will be created for the storage cluster.

[0102] It should be noted that Figure 3 Application scenarios of the metadata storage method provided in the embodiments of the present application Figure 2 ,like Figure 3As shown in the figure, when the storage service is started, it will first be registered with the service registration center so that the cluster manager can discover and manage it. After the storage service is started, it can receive metadata forwarded by the cluster manager to achieve persistent storage of metadata.

[0103] It should be understood that each storage cluster has its own weight coefficient to determine the amount of data that the storage cluster can carry. The request distributor will confirm the storage cluster to which the metadata belongs based on the weight coefficient and send the metadata to the storage cluster to be stored and processed.

[0104] In a specific example, it is assumed that there are two storage clusters, storage cluster 1 and storage cluster 2, where storage cluster 1 is an existing cluster and storage cluster 2 is a newly created cluster. Storage cluster 1 and storage cluster 2 each include at least one storage service, and the storage services in storage cluster 1 and storage cluster 2 have been registered in the service registration center.

[0105] The request distributor creates cluster manager 1 for storage cluster 1 and cluster manager 2 for storage cluster 2 through the service discovery mechanism. The cluster manager is responsible for initializing, updating, and removing storage services. At the same time, the cluster weight manager dynamically sets the corresponding weight coefficients for storage cluster 1 and storage cluster 2, and notifies the cluster selector of the weight changes of the storage clusters. Next, the cluster selector is responsible for determining the storage cluster to which the metadata belongs and forwarding it based on the weight coefficients.

[0106] Specifically, Figure 4 Schematic diagram of the metadata storage method provided in the embodiment of the present application Figure 2 ,like Figure 4 As shown, sending the metadata of the file system uploaded by the user through the file system client to the corresponding storage cluster includes the following steps:

[0107] S401, metadata first arrives at the request distributor;

[0108] S402, determining whether a target storage cluster is specified in advance for the metadata;

[0109] If not, execute S403; if so, execute S404.

[0110] S403, based on the weight coefficient, determine the target storage cluster through the cluster selector, and execute S405;

[0111] S404, searching for a pre-specified target storage cluster through a cluster selector, and executing S405;

[0112] S405, forwarding the metadata to a cluster manager corresponding to the target storage cluster;

[0113] S406: Send the metadata to the storage service of the target storage cluster through the cluster manager for storage processing.

[0114] It can be seen that when there is too much metadata waiting to be stored, the purpose of horizontal expansion can be achieved by recreating a new storage cluster. In the process of creating a new storage cluster, the storage service of the existing storage cluster will not be interrupted, and the metadata can still be stored and processed, thereby ensuring the storage efficiency of the metadata. Afterwards, the metadata that needs to be stored is apportioned by dynamically setting weight coefficients for the storage clusters to avoid some storage clusters being idle while other storage clusters are overloaded, thus achieving load balancing between storage clusters and improving the resource utilization of the storage clusters.

[0115] Figure 5 A schematic diagram of the structure of the metadata storage device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the metadata storage device 500 includes: an acquisition module 501, a determination module 502, a creation module 503, and a storage module 504;

[0116] Wherein, the acquisition module 501 is used to acquire metadata of the file system waiting to be stored;

[0117] The determination module 502 is used to determine a first storage cluster and a total amount of the metadata; wherein the first storage cluster represents an existing cluster for storing metadata;

[0118] A creation module 503 is used to create a new second storage cluster for storing metadata if the total data volume is greater than a preset threshold;

[0119] The storage module 504 is used to dynamically set a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule; and send the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient.

[0120] Optionally, the first storage cluster includes at least one first storage service, and the second storage cluster includes at least one second storage service;

[0121] The determination module 502 is further configured to determine a total data storage capacity of the first storage service;

[0122] The creation module 503 further includes: a registration module 505, which is used to register the second storage service if the total data volume is greater than the total data storage volume, and form the second storage cluster based on the registered second storage service.

[0123] Optionally, the storage module 504 further includes: a calculation module 506 and a setting module 507.

[0124] The calculation module 506 is used to calculate in real time a first storable data volume of a first storage service included in the first storage cluster, and a second storable data volume of a second storage service included in the second storage cluster;

[0125] The setting module 507 is used to dynamically set a weight coefficient for the first storage cluster and the second storage cluster based on the first storable data volume and the second storable data volume.

[0126] Optionally, the metadata storage device 500 further includes: a monitoring module 508, configured to monitor the first storage cluster and the second storage cluster;

[0127] The creation module 503 is further configured to create a cluster manager for the first storage cluster and / or the second storage cluster when detecting that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are online;

[0128] The metadata storage device 500 also includes: a clearing module 509, which is used to clear the cluster manager of the first storage cluster and / or the second storage cluster when it is detected that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are both offline.

[0129] Optionally, the monitoring module 508 further includes: a sending module 510, configured to send a heartbeat detection signal to a first storage service in the first storage cluster and a second storage service in the second storage cluster;

[0130] The determination module 502 is also used to determine that the first storage service and / or the second storage service is in an online state if a heartbeat response is received from the first storage service and / or the second storage service based on the heartbeat detection signal; if the heartbeat response is not received, determine that the first storage service and / or the second storage service is in an offline state.

[0131] Optionally, the storage module 504 is further configured to send the metadata to the target storage cluster for storage processing if a target storage cluster is specified in advance for the metadata; wherein the target storage cluster is the first storage cluster or the second storage cluster;

[0132] The determination module 502 is further configured to determine a target storage cluster from the first storage cluster or the second storage cluster based on the weight coefficient if no target storage cluster is specified in advance for the metadata;

[0133] The storage module 504 is further configured to send the metadata to a target storage cluster for storage processing.

[0134] Optionally, the determination module 502 is also used to determine that the first storage cluster and / or the second storage cluster is in a data saturation state if the weight coefficient of the first storage cluster and / or the second storage cluster is zero, and no longer send metadata to the first storage cluster and / or the second storage cluster.

[0135] The metadata storage device provided in the embodiment of the present application can be used to execute the metadata storage method in any of the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0136] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0137] Figure 6 The hardware structure diagram of the electronic device provided in this application. Figure 6 As shown, the electronic device 600 includes:

[0138] Processor 601 and memory 602;

[0139] Memory stores computer-executable instructions;

[0140] The processor executes the computer execution instructions stored in the memory 602, so that the electronic device executes the metadata storage method as described above.

[0141] It should be understood that the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The memory 602 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk, or an optical disk.

[0142] The electronic device provided in the embodiment of the present application can be used to execute the metadata storage method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0143] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the metadata storage method as described above.

[0144] The embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to implement the metadata storage method as described above.

[0145] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0146] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0147] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0148] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0149] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0150] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0153] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A metadata storage method, characterized in that: include: Obtaining metadata of a file system waiting to be stored, and determining a first storage cluster; wherein the first storage cluster represents an existing cluster for storing metadata; Determining a total data volume of the metadata, and if the total data volume is greater than a preset threshold, creating a new second storage cluster for storing the metadata; Dynamically setting a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule; and sending the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient.

2. The method according to claim 1, characterized in that The first storage cluster includes at least one first storage service, and the second storage cluster includes at least one second storage service; If the total data volume is greater than a preset threshold, creating a new second storage cluster for storing metadata includes: Determine the total data storage capacity of the first storage service; If the total data volume is greater than the total data storage volume, the second storage service is registered, and the second storage cluster is formed based on the registered second storage service.

3. The method according to claim 1, characterized in that The dynamically setting a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule includes: Calculating in real time a first storable data volume of a first storage service included in the first storage cluster, and a second storable data volume of a second storage service included in the second storage cluster; Based on the first storable data volume and the second storable data volume, a weight coefficient is dynamically set for the first storage cluster and the second storage cluster.

4. The method according to claim 2, characterized in that: The method further comprises: Monitoring the first storage cluster and the second storage cluster; When detecting that a first storage service in the first storage cluster and / or a second storage service in the second storage cluster are online, creating a cluster manager for the first storage cluster and / or the second storage cluster; When it is detected that the first storage service in the first storage cluster and / or the second storage service in the second storage cluster are both in an offline state, the cluster manager of the first storage cluster and / or the second storage cluster is cleared.

5. The method according to claim 4, characterized in that The monitoring of the first storage cluster and the second storage cluster further includes: Sending a heartbeat detection signal to a first storage service in the first storage cluster and a second storage service in the second storage cluster; If a heartbeat response is received from the first storage service and / or the second storage service based on the heartbeat detection signal, it is determined that the first storage service and / or the second storage service is in an online state; if the heartbeat response is not received, it is determined that the first storage service and / or the second storage service is in an offline state.

6. The method according to claim 1, characterized in that The step of sending the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient includes: If a target storage cluster is specified in advance for the metadata, the metadata is sent to the target storage cluster for storage processing; wherein the target storage cluster is the first storage cluster or the second storage cluster; If a target storage cluster is not specified in advance for the metadata, a target storage cluster is determined from the first storage cluster or the second storage cluster based on the weight coefficient; and the metadata is sent to the target storage cluster for storage processing.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: If the weight coefficient of the first storage cluster and / or the second storage cluster is zero, it is determined that the first storage cluster and / or the second storage cluster is in a data saturation state, and metadata is no longer sent to the first storage cluster and / or the second storage cluster.

8. A metadata storage device, characterized in that: include: An acquisition module, used to acquire metadata of a file system waiting to be stored; A determination module, configured to determine a first storage cluster and a total amount of data of the metadata; wherein the first storage cluster represents an existing cluster for storing metadata; A creation module for creating a new second storage cluster for storing metadata if the total amount of data is greater than a preset threshold; The storage module is used to dynamically set a weight coefficient for the first storage cluster and the second storage cluster based on a preset rule; and send the metadata to the first storage cluster or the second storage cluster for storage processing according to the weight coefficient.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the metadata storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the metadata storage method according to any one of claims 1 to 7.

Images