Metadata Processing Method, Computer Device, Readable Storage Medium and Program Product

Abstract

The present application discloses a metadata processing method, a computer device, a readable storage medium, and a program product, relating to the field of computer technologies, including processing metadata based on different controller operation types and the service weights of multiple initial controllers; the controller operation types include: a controller addition operation type and a controller removal operation type, and different metadata processing schemes are set for different controller operation types, supporting fast metadata migration in multiple scenarios, solving the technical problem of a long recovery time in the related art of a single metadata recovery service, and achieving the technical effect of improving the reliability of the metadata service.

Description

Technical Field

[0001] The present application relates to the field of computer technologies, and in particular, to a metadata processing method, a computer device, a readable storage medium, and a program product. Background Art

[0002] When a metadata cluster fails, other metadata services in use take over the failed metadata service to resume the cluster's external services. During this recovery period, the taking-over metadata service does not provide external services, that is, it does not respond to client requests. Therefore, the length of the recovery time directly affects the user experience.

[0003] In related technologies, the traditional file system metadata service is provided by a single controller, which has a single point of failure, and all IOs (input / outputs) wait for the single metadata service to recover. Therefore, the existing metadata service cannot meet the requirements for metadata recovery time in a failure drift scenario, resulting in a decline in the reliability of the metadata service. Summary of the Invention

[0004] The present application provides a metadata processing method, a computer device, a readable storage medium, and a program product to solve the technical problem of the long recovery time of a single metadata recovery service in related technologies.

[0005] The present application provides a metadata processing method, including: obtaining a controller operation type; obtaining the service weight of an initial controller; in response to the controller operation type being a controller addition operation, determining a first traffic volume that matches the service weight of the initial controller, and processing the newly added metadata and the metadata of the initial controller based on the first traffic volume, where the first traffic volume represents the traffic volume that the newly added controller needs to bear; in response to the controller operation type being a controller removal operation, selecting a non-removal controller from multiple initial controllers, determining a second traffic volume that matches the service weight of the non-removal controller, and processing the metadata to be removed of the controller to be removed based on the second traffic volume, where the second traffic volume represents the traffic volume that the non-removal controller needs to bear.

[0006] The present application also provides a computer device, including: a memory for storing a computer program; a processor for implementing the steps of the metadata processing method in the following embodiments when executing the computer program.

[0007] Obtain the controller operation type; obtain the service weight of the initial controller; in response to the controller operation type being a controller addition operation, determine a first service volume that matches the service weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first service volume; wherein, the first service volume represents the service volume that the newly added controller needs to bear; in response to the controller operation type being a controller removal operation, select non-removal controllers from multiple initial controllers, determine a second service volume that matches the service weight of the non-removal controllers, and process the metadata to be removed of the controller to be removed based on the second service volume; wherein, the second service volume represents the service volume that the non-removal controllers need to bear.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata processing method in the following implementation manners are realized.

[0009] Obtain the controller operation type; obtain the service weight of the initial controller; in response to the controller operation type being a controller addition operation, determine a first service volume that matches the service weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first service volume; wherein, the first service volume represents the service volume that the newly added controller needs to bear; in response to the controller operation type being a controller removal operation, select non-removal controllers from multiple initial controllers, determine a second service volume that matches the service weight of the non-removal controllers, and process the metadata to be removed of the controller to be removed based on the second service volume; wherein, the second service volume represents the service volume that the non-removal controllers need to bear.

[0010] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the metadata processing method in the following implementation manners are realized.

[0011] Obtain the controller operation type; obtain the service weight of the initial controller; in response to the controller operation type being a controller addition operation, determine a first service volume that matches the service weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first service volume; wherein, the first service volume represents the service volume that the newly added controller needs to bear; in response to the controller operation type being a controller removal operation, select non-removal controllers from multiple initial controllers, determine a second service volume that matches the service weight of the non-removal controllers, and process the metadata to be removed of the controller to be removed based on the second service volume; wherein, the second service volume represents the service volume that the non-removal controllers need to bear.

[0012] The metadata processing method provided by this application processes metadata based on the controller operation type and the business weights of multiple initial controllers. Among them, when the controller operation type is a controller addition operation, the first business volume that matches the business weight of the initial controller is determined, and the new metadata and the metadata of the initial controller are processed based on the first business volume. Here, the first business volume represents the business volume that the newly added controller needs to bear. When the controller operation type is a controller removal operation, non-removal controllers are selected from multiple initial controllers, the second business volume that matches the business weight of the non-removal controllers is determined, and the metadata to be removed of the controller to be removed is processed based on the second business volume. Here, the second business volume represents the business volume that the non-removal controllers need to bear. Therefore, it supports fast metadata migration in multiple scenarios, solves the technical problem of long recovery time in the related art of a single metadata recovery service, and achieves the technical effect of improving the reliability of the metadata service. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] To more clearly illustrate the embodiments of this application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0014] Figure 1 It is a schematic structural diagram of a metadata processing system provided by an embodiment of this application;

[0015] Figure 2 It is a schematic flowchart of a metadata processing method provided by an embodiment of this application;

[0016] Figure 3 It is a schematic flowchart of a metadata processing method provided by another embodiment of this application;

[0017] Figure 4 It is a schematic flowchart of a metadata processing method provided by yet another embodiment of this application;

[0018] Figure 5 It is a schematic structural diagram of a metadata processing device provided by an embodiment of this application;

[0019] Figure 6 It is an internal structure diagram of a computer device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0021] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0023] As Figure 1 shown, the metadata processing method of the present application is applied to a metadata processing system, and the metadata processing system specifically includes: a host, multiple controllers (controller A and controller B shown in the figure), cluster management (cluster status manager), and a NAS pool.

[0024] The metadata processing system of the present application is implemented based on a centralized NAS architecture. NAS (Network Attached Storage) is a file system sharing device connected to an IP (Internet Protocol Address) local area network. It separates storage devices from servers and allows users to access data through the network.

[0025] The host refers to the central component with an intelligent network, which is responsible for running the software and hardware of all network devices and managing all data, files, and information in the network.

[0026] The controller is an indispensable part of an automated system. By receiving input signals from sensors or other devices and generating corresponding output signals through preset control strategies or algorithms, precise control of the controlled object can be achieved.

[0027] As shown in the figure, a metadata service is set on each controller in this application. Each file system has a metadata service on each controller (a service that provides access functions for metadata, a system or platform for managing and operating metadata). The metadata services on all controllers form a cluster. The failure of the metadata service on a certain controller does not affect the overall availability of the file system. The metadata service has 7 states: initial (start) / loading (loading) / normal (normal) / unloading (unloading) / unloaded (unloaded) / offline (offline) / deleted (deleted).

[0028] The controller also includes NAS network settings, NAS sharing protocol service settings, data space management settings, etc. By setting the NAS network, NAS sharing protocol service, and data space management, it is used to achieve data sharing and file management of network attached storage devices.

[0029] Cluster management (Monitor) refers to the manager of the cluster status, maintaining the status of the entire cluster and jointly forming a cluster view with other cluster status managers. The cluster manager can manage the NAS file system view, metadata service view, and Internet protocol address view, and use cluster interconnection links such as NTB / FC / RoCE for configuration synchronization and message passing. Through the PAXOS consistency algorithm, the primary controller is selected.

[0030] The NAS (Network Attached Storage) pool is a device for storing and sharing data. It is connected to the network in the local area network and provides file access services. The NAS pool integrates the storage resources of multiple NAS devices into a unified storage pool, forming a logically centralized managed storage system. After pooling the storage resources, the administrator can manage and allocate storage resources through a unified interface without caring about the specific physical devices. This can improve the utilization rate of storage resources, reduce management costs, and make the storage system more flexible and scalable.

[0031] As Figure 2 shown, an embodiment of this application provides a metadata processing method, which specifically includes the following steps:

[0032] Step 101: Obtain the controller operation type.

[0033] Step 102: Obtain the service weight of the initial controller.

[0034] Specifically, the operation types of the controllers include controller addition operations and controller removal operations. In this application, a cluster including multiple controllers is set up. A controller addition operation refers to adding a new controller to the cluster, and a controller removal operation refers to removing a controller from the cluster. The initial controllers refer to each controller in the current cluster.

[0035] Controller information can be obtained. The controller information includes at least one of controller load information, controller failure rate, and controller latency information, etc. The controller information is input into the service weight calculation formula to obtain the service weights of the initial controllers.

[0036] The controller load information can include the number of processor cores. The number of processor cores refers to the number of independent physical processing units inside the CPU, and each core can execute computing tasks; the number of processes that the controller is running and the number of processes that the controller is waiting to run.

[0037] The controller failure rate can be the frequency of failures that the controller has occurred within a certain period of history, which can reflect the health status of the controller.

[0038] For the controller latency information, the controller latency information can refer to the time required for data to travel from the controller to the receiving data controller. The latency information is one of the core metrics for measuring network performance.

[0039] Specifically, the calculation formula for the service weight is as follows:

[0040]

[0041] Among them, Score A represents the service weight of the initial controller A, X represents the weight of the controller load impact, CPU loadpct represents the load of the initial controller A, which is the ratio of the sum of the processes that the initial controller A is running and the processes waiting to run to the number of processor cores corresponding to the initial controller A. Y represents the weight of the controller historical failure rate impact, HFR represents the failure rate of the initial controller A, which is the ratio of the number of failures that the initial controller A has occurred within the preset historical period to the preset period. Z represents the weight of the controller latency impact, IO latency represents the average latency corresponding to the read and write operations of the initial controller A. H and M represent adjustment parameters, and t represents a percentage.

[0042] Exemplarily, for example, the number of CPU cores of controller A is 40, and the number of processes that are running and waiting to run is 10. Then the load of the initial controller A is 0.25. Assuming that controller A has had 2 failures within 100 days, then the controller failure rate is 0.02.

[0043] The service weight of a controller reflects the service load situation of the controller. The lower the service weight of the controller, the less service volume the controller should carry. Conversely, the higher the service weight of the controller, the higher the service volume the controller should carry.

[0044] By comprehensively considering the controller load, controller failure rate, and controller latency (average IO latency), determine the service weight of each controller, which is subsequently used to calculate the service volume carried by the controller, and can maximize the load balancing of the entire cluster.

[0045] In the operation types of this application, different metadata processing schemes are provided for different controller operation types.

[0046] Step 1021: In response to the controller operation type being a controller addition operation, determine the first service volume that matches the service weight of the initial controller, and process the new metadata and the metadata of the initial controller based on the first service volume; wherein, the first service volume represents the service volume that the newly added controller needs to carry.

[0047] Specifically, obtain the number of newly added controllers, the total service weight of multiple initial controllers, and calculate the service volume that the newly added controller needs to carry using the first service volume calculation formula; perform data migration on the new metadata and the metadata of the initial controller according to the service volume that the newly added controller needs to carry, and balance the distribution of the new metadata and the metadata of the initial controller among the initial controllers and the newly added controllers.

[0048] Among them, the first service volume calculation formula can be as follows:

[0049]

[0050] Among them, W inM represents the service volume that the newly added controller M needs to carry, Count newController represents the number of newly added controllers, Score i represents the score of the initial controller i, ΣCount curController Score represents the total service weight of multiple initial controllers, W i represents the current service volume of the initial controller i.

[0051] In an embodiment, the metadata processing method further includes: obtaining the current service volume of the initial controller; inputting the service weight and the current service volume of the initial controller into the migration service volume calculation formula to obtain the migration service volume of the initial controller; wherein, the migration service volume of the initial controller represents the service volume that the initial controller needs to migrate to the newly added controller.

[0052] Among them, the migration service volume calculation formula is expressed as:

[0053]

[0054] Among them, W outA represents the migration traffic volume of the initial controller A, and Score A represents the service weight of the initial controller A, and ΣCount curController Score represents the total service weight of multiple initial controllers; W A represents the current traffic volume of the initial controller A.

[0055] When a controller is added, the cluster status manager will initialize a new metadata service, that is, configure the metadata service on the newly added controller, and bind the corresponding metadata service to the newly added controller according to the location information of the newly added controller. At this time, the cluster status manager will trigger dynamic load balancing, and calculate the traffic volume that needs to be borne on the newly added controller and the traffic volume that each initial controller in the cluster needs to migrate to the newly added controller respectively through the first traffic volume calculation formula and the migration traffic volume calculation formula. In this way, after a new controller is added to the cluster, load balancing between the newly added controller and the initial controllers in the cluster can be achieved.

[0056] Step 1022: In response to the controller operation type being a controller removal operation, select non-removal controllers from multiple initial controllers, determine the second traffic volume that matches the service weight of the non-removal controllers, and process the metadata to be removed of the controller to be removed based on the second traffic volume; among them, the second traffic volume represents the traffic volume that the non-removal controllers need to bear.

[0057] Specifically, determine the service weight of the non-removal controllers based on the service weights of multiple initial controllers; calculate the traffic volume that the non-removal controllers need to bear based on the service weight of the non-removal controllers, the current traffic volumes of multiple initial controllers, and the non-removal controller traffic volume calculation formula; perform data migration on the metadata of multiple initial controllers based on the traffic volume that the non-removal controllers need to bear, so that the metadata of multiple initial controllers is evenly distributed on the non-removal controllers.

[0058] Specifically, the non-removal controller traffic volume calculation formula is expressed as:

[0059]

[0060] Among them, W inB represents the traffic volume that the non-removal controller B needs to bear, and Score B represents the service weight of the non-removal controller B, and ΣCount avaController Score represents the total service weight of multiple non-removal controllers; W CIndicates the traffic volume to be removed.

[0061] A non-removing controller refers to a controller that does not remove the cluster. When a controller in the cluster is removed, load balancing is triggered. According to the service weights of each non-removing controller, the metadata carried on the removing controller needs to be distributed to the non-removing controllers in the cluster to balance the load among the non-removing controllers that have received the metadata from the removing controller.

[0062] In this application, according to the different scenario changes of the controller, following the calculation formulas corresponding to different scenarios, both the metadata processing efficiency of scenario switching is considered and the data security can be ensured.

[0063] In one embodiment, before obtaining the controller operation type, it further includes: creating a file system, performing metadata service initialization based on the number of controllers in the current cluster, configuring metadata services for each controller in the cluster, creating a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service statuses; in response to the completion of the creation of the file system view and the metadata service view, the host issues read / write requests, mounts the file system with the Internet protocol address corresponding to the controller, and creates an Internet protocol address view. The Internet protocol address map includes the Internet protocol addresses bound to the network interfaces of multiple controllers to enable the controller to access the file system mounted by the controller in response to the read / write requests issued by the host. By creating the file system view, the metadata service view, and the Internet protocol address view, the metadata service status, the Internet protocol address binding information, etc. can be obtained in a timely manner. By adopting multiple metadata services, the metadata service view is maintained by the cluster status manager to control the start and stop of the metadata services.

[0064] In practical applications, users perform file system configuration through the GUI (Graphical User Interface) / CLI (Command Line Interface). The Mon (Cluster State Manager) selects to initialize the MDS (Metadata Service) based on the number of controllers in the current cluster. According to the location information of N controllers, the initialized MDS is divided into instance0 to instance(N - 1). Under normal conditions, instanceX (instance X) is on the X controller in the cluster. Since the MDS is the implementation of the file system, it does not require users to specify and is bound to the location information of the controllers. Therefore, the metadata service only has the properties of the currently running controllers, and the belonging of the specified controllers is calculated in real time through the location information. That is, users only care about creating one file system, but how many metadata services this file system is carried by is an internal implementation and is not externally displayed. As long as the instance number is fixed internally, for example, instance is on the first controller, and instancen - 1. During the configuration process, the state transition of the metadata service involves initial (start) -> loading (loading) -> normal (normal).

[0065] Users perform IP configuration through the GUI / CLI, and the parameters include the specified controller and network interface, that is, the preferred controller and network interface information of this IP (Internet Protocol Address) under normal conditions. The Cluster State Manager adds the IP to the specified port on the specified controller based on these two parameters of the controller and the network interface, and then notifies all controllers through broadcasting to update the Internet Protocol Address view. The IP state transition during the configuration process includes initial (start) -> loading (loading) -> normal (normal). When there is a network interface failure or a controller failure, the IP configured on the network interface is configured by the Cluster State Manager on the network interface of another controller to achieve the availability of services in the failure scenario. The IP includes two attributes, the preferred controller and the bound controller, that is, the running controller specified during configuration and the currently running controller. The IP has 5 states: initial (start) / loading (loading) / normal (normal) / unloading (unloading) / unloaded (unloaded). Of course, the Cluster State Manager will also notify all controllers in the MDS cluster of the updated metadata service view.

[0066] Data migration is involved both when the controller operation type is the controller addition operation type and when the controller operation type is the controller removal operation type. Here, data migration includes removing the metadata to be removed and the corresponding Internet protocol address on the controller to be removed, and receiving the removed metadata and setting the Internet protocol address corresponding to the removed metadata on the receiving controller corresponding to the controller to be removed. That is, when performing metadata migration, both the metadata and its corresponding Internet protocol address need to be unbound from the controller to be removed and rebound to the receiving controller.

[0067] Specifically, data migration includes: in response to receiving a data migration instruction, obtaining a metadata service view and an Internet protocol address view; traversing the Internet protocol address view, marking the Internet protocol address corresponding to the metadata to be removed as a stopped Internet protocol address, and marking the Internet protocol address corresponding to the received removed metadata as a started Internet protocol address; traversing the metadata service view, marking the metadata to be removed as stopped metadata, and marking the received removed metadata as started metadata; stopping the running program of the Internet protocol address marked as the stopped Internet protocol address; in response to successfully stopping the running program of the Internet protocol address marked as the stopped Internet protocol address, stopping the metadata service corresponding to the marked stopped metadata; in response to successfully stopping the metadata service corresponding to the marked stopped metadata, starting the metadata service corresponding to the marked started metadata; in response to successfully starting the metadata service corresponding to the marked started metadata, starting the running program of the Internet protocol address marked as the started Internet protocol address.

[0068] Such as Figure 3As shown, specifically, stop running the corresponding unbinding operation, start the corresponding binding operation, traverse the Internet Protocol address view, mark the IP that needs to be started and the IP that needs to be stopped, traverse the metadata service view, mark the MDS (metadata service) that needs to be started and the MDS that needs to be stopped, traverse all IPs that need to be stopped, and perform a stop action on the corresponding currently running controller until all IPs that need to be stopped have been stopped, the stop action can be concurrent, and continue after completion, otherwise wait; traverse all MDSs that need to be stopped, and perform a stop action on the corresponding currently running controller until all MDSs that need to be stopped have been stopped, the stop action can be concurrent, and continue after completion, otherwise wait; traverse all MDSs that need to be started, and perform a start action on the controller that needs to be started until all MDSs that need to be started are started, the start action can be concurrent, and continue after completion, otherwise wait; traverse all IPs that need to be started, and perform a start action on the controller that needs to be started until all IPs that need to be started are started, the start action can be concurrent, and continue after completion, otherwise wait; That is, you need to start the metadata service first, then configure the IP, prepare the metadata service first, and then provide IP access to the outside world. If the IP is configured first and the metadata service is not configured yet, an error will be reported.

[0069] In one embodiment, the controller operation type also includes a controller failure operation type, and the metadata processing method also includes: in response to the controller operation type being a controller failure operation, selecting a non-faulty controller that meets preset conditions from multiple initial controllers to take over the metadata of the faulty controller; wherein the preset conditions include the business weights of other non-faulty controllers whose business weights are lower than the target number.

[0070] The non-faulty controller with the lowest business weight or the target number of non-faulty controllers with low business weight can be directly selected to take over the metadata of the faulty controller. The selection of the number of non-faulty controllers here depends on the business volume of the metadata of the faulty controller and the business acceptance capacity of the non-faulty controller with low business weight.

[0071] Controller failure is different from controller removal. Controller removal refers to a long-term failure, and the unusable controller will be removed, while controller failure refers to a short-term failure, such as controller disconnection, controller power outage, etc. The controller is unavailable for a short period of time. In this application, the old controller failure will not remove the metadata service of the failed controller. The controller with the smallest business weight will be selected to completely take over the business of the failed controller. The disadvantage is that the business pressure is uneven for a short period of time, but considering that the controller failure is not a long-term stable state, it is acceptable. The advantage of this is to increase the speed of business migration and avoid splitting the business.

[0072] In one embodiment, the controller operation type further includes controller recovery, and the metadata processing method further includes: in response to the controller operation type being the controller recovery operation type, determining a target recovery controller corresponding to the metadata service to be recovered, and binding the metadata service to be recovered to the target recovery controller based on the target recovery controller.

[0073] Controller recovery is an operation reverse to controller failure. The cluster status manager uses the preferred binding controller of the metadata service to be recovered as the target recovery controller, binds the metadata service to be recovered to the target recovery controller, and broadcasts the updated metadata service view to all controllers. Each controller determines whether to start or stop the corresponding metadata service based on the current running status of the metadata service and the latest view.

[0074] In one embodiment, the metadata processing method further includes: obtaining the controller network interface status type, and determining the binding controller of the Internet protocol address corresponding to the network interface according to the controller network interface status type; wherein, when the controller network interface status type is network interface failure, obtaining the order of the controller with the network interface failure in the chassis, determining the network interface of the binding controller according to the order of the controller in the chassis, and binding the network interface of the binding controller to the Internet protocol address corresponding to the failed network interface. When the controller network interface status type is network interface recovery, the cluster status manager searches in the Internet protocol address view for the target controller corresponding to the Internet protocol address of the network interface to be recovered. The target controller is the specified running controller and the target controller satisfies that the currently running Internet protocol address corresponding to the network interface to be recovered does not belong to the target controller, and binds the Internet protocol address corresponding to the recovered network interface to the target controller.

[0075] After a network interface fails, it is necessary to select another network interface to configure the Internet protocol address. The metadata service defaults to searching for the port at the same position as the binding controller's network interface according to the order of the controllers in the chassis, and then broadcasts the Internet protocol address view to all controllers. Each controller selects to stop or start the IP according to the binding controller information of the IP in the view, that is, if the binding controller is not this controller but the IP is running on this controller, then stop the IP; if the binding controller is this controller but the IP is not running on this controller, then start the IP, that is, the target controller is the specified running controller and the target controller satisfies that the currently running Internet protocol address corresponding to the network interface to be recovered does not belong to the target controller.

[0076] Such as Figure 4As shown, in order to avoid data inconsistency, the switching order needs to be ensured as follows: first, the cluster status manager will stop the running program of the IP with the network port fault, and return to the cluster status manager that the running program of the IP with the network port fault has stopped successfully. In response to the cluster status manager receiving the running program of the IP with the network port fault has stopped successfully, the running program of the IP of the target controller is started, and the running program of the IP of the target controller is returned to the cluster status manager that the running program of the IP has started successfully.

[0077] After the network port of a controller is restored, Mon traverses the complete Internet Protocol address view, searches for the IP of the preferred controller that belongs to the controller and the currently running controller that does not belong to the controller, updates the bound controller in the Internet Protocol address view to the controller, and broadcasts the Internet Protocol address view. Each controller chooses to stop or start the IP based on the bound controller information in the view.

[0078] In one embodiment, the present application further includes obtaining the storage pool status type corresponding to the controller, and processing the metadata service corresponding to the controller according to the storage pool status type; wherein, when the storage pool status type corresponding to the controller is that the storage pool is offline, a service interruption instruction is triggered to prohibit the operation of the metadata service corresponding to the offline storage pool. When the storage pool status type corresponding to the controller is that the storage pool is restored online, a service recovery start instruction is triggered to restore and start the metadata service on the controller corresponding to the file system on the restored storage pool, and a metadata service status switching instruction is triggered to switch the metadata service status to normal operation of the metadata service based on the metadata service status switching instruction.

[0079] After the storage pool is offline, the metadata service on the storage pool cannot be started normally no matter which controller it is migrated to. At this time, the business is interrupted and the metadata service does not need to be migrated. In other words, the metadata on the controller corresponding to the metadata service is not migrated, and the metadata service status switches to normal->unloading->unloaded->offline.

[0080] After the storage pool is restored and online, the MDS cluster corresponding to the file system on the storage pool needs to be restored and started, triggering the state machine switch. The MDS state switches to offline->loading->normal. By starting the metadata service instance on each controller to form a metadata service cluster, the single point failure problem of the metadata service is solved. By controlling the drift order of the metadata service (MDS) and IP (Internet Protocol address), the efficiency and security of business drift in scenarios such as failures are solved.

[0081] In a specific implementation, the metadata processing method of the present application may be shown in the following steps:

[0082] S1: When the service request is a metadata write request, obtain the size of the metadata to be written to the controller and the size of the remaining data storage space corresponding to the cluster.

[0083] S2: Compare the size of the remaining data storage space corresponding to the cluster with the size of the metadata to be written to the controller to obtain a comparison result.

[0084] S21: When the size of the remaining data storage space corresponding to the cluster is greater than or equal to the size of the metadata to be written to the controller, it is considered that no new controller needs to be added. Obtain the initial number of controllers in the cluster. Use the initial number of controllers in the cluster as the divisor and the size of the metadata to be written to the controller as the dividend to calculate the average metadata to be written for each controller.

[0085] S212: Obtain the size of the remaining data storage space of each initial controller in the cluster. Compare the average metadata to be written for each controller with the size of the remaining data storage space of the controller corresponding to the initial controller to obtain a comparison result.

[0086] S2121: If the size of the remaining data storage space of any initial controller is less than the average metadata to be written for each controller, determine the allocation ratio based on the size of the remaining data storage space of each initial controller. Based on the allocation ratio, allocate the metadata to be written to each initial controller in the cluster.

[0087] Here, the size of the remaining data storage space of the initial controller that is less than the average metadata to be written for each controller can be used as the first allocated storage space size. Select the target metadata to be written that is less than or equal to the first allocated storage space size and allocate it to the controller corresponding to the first allocated storage space size. Calculate the difference between the metadata to be written and the target metadata to be written, and use this difference as the metadata to be allocated. Obtain the initial controllers in the cluster that have not been allocated the metadata to be written and their corresponding service weights. Allocate a higher allocation ratio to the initial controller with a higher service weight among the initial controllers that have not been allocated the metadata to be written. Based on the allocation ratio, allocate the metadata to be allocated to the initial controllers that have not been allocated the metadata to be written. In this way, the size of the remaining data storage space of each initial controller can be fully considered, and the metadata to be written can be evenly distributed on each initial controller.

[0088] S2122: If the size of the remaining data storage space of each initial controller is greater than the average metadata to be written for each controller, evenly divide the metadata to be written among each controller according to the average metadata to be written for each controller.

[0089] S22: When the size of the remaining data storage space corresponding to the cluster is smaller than the metadata size of the controller to be written, it is considered that a new controller needs to be added, and the cluster state manager (Mon) triggers the addition of dynamic load balancing.

[0090] Determine the number of new controllers based on the metadata size of the controller to be written and the size of the remaining data storage space corresponding to the cluster.

[0091] S22: In response to triggering the addition of dynamic load balancing, obtain controller load information, controller failure rate, and controller delay information, and determine the business acceptance weight of each initial controller based on the controller load information, controller failure rate, and controller delay information.

[0092] S23: Calculate the first traffic volume based on the business acceptance weight of each initial controller.

[0093] S24: Process the new metadata and the metadata of the initial controller based on the first traffic volume.

[0094] Before obtaining the metadata size of the controller to be written and the size of the remaining data storage space corresponding to the cluster, it is also possible to determine whether the controller has been in a long-term failure, the storage pool status, the controller network interface status, etc., and perform different metadata processing operations according to different statuses.

[0095] An embodiment of the present application provides a metadata processing device, and the metadata processing device is specifically as Figure 5 shown. The metadata processing device includes: an acquisition module 20 and a processing module 21.

[0096] The acquisition module 20 is used to acquire the controller operation type and acquire the business weight of the initial controller;

[0097] The processing module 21 is used to, in response to the controller operation type being a controller addition operation, determine a first traffic volume matching the business weight of the initial controller, and process the new metadata and the metadata of the initial controller based on the first traffic volume; wherein, the first traffic volume represents the traffic volume that the new controller needs to carry; in response to the controller operation type being a controller removal operation, select non-removal controllers from multiple initial controllers, determine a second traffic volume matching the business weight of the non-removal controllers, and process the metadata to be removed of the controller to be removed based on the second traffic volume; wherein, the second traffic volume represents the traffic volume that the non-removal controllers need to carry.

[0098] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: Obtaining the service weight of the initial controller includes: obtaining the controller information of the initial controller; wherein, the controller information includes at least one of controller load information, controller failure rate, and controller delay information; inputting the controller information into the service weight calculation formula to obtain the service weight of the initial controller.

[0099] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: The service weight calculation formula is expressed as:

[0100]

[0101] Wherein, Score A represents the service weight of the initial controller A, X represents the weight of the controller load impact, CPU loadpct represents the load of the initial controller A, which is the ratio of the sum of the processes running on the initial controller A and the processes waiting to run to the number of processor cores corresponding to the initial controller A, Y represents the weight of the controller historical failure rate impact, HFR represents the failure rate of the initial controller A, which is the ratio of the number of failures of the initial controller A within the historical preset period to the preset period, Z represents the weight of the controller delay impact, IO latency represents the average delay corresponding to the read and write operations of the initial controller A, H and M represent adjustment parameters, and t represents a percentage.

[0102] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: Determining the first traffic volume matching the service weight of the initial controller, and processing the new metadata and the metadata of the initial controller based on the first traffic volume includes: obtaining the number of new controllers, the total service weight of multiple initial controllers, and calculating the traffic volume that the new controllers need to bear according to the first traffic volume calculation formula; performing data migration on the new metadata and the metadata of the initial controller according to the traffic volume that the new controllers need to bear to balance the distribution of the new metadata and the metadata of the initial controller among the initial controllers and the new controllers.

[0103] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: The first traffic volume calculation formula is expressed as:

[0104]

[0105] Wherein, W inM represents the traffic volume that the new controller M needs to bear, Count newController represents the number of new controllers, Score iRepresents the score of the initial controller i, ΣCount curController Score represents the total service weight of multiple initial controllers, W i Represents the current service volume of the initial controller i.

[0106] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: The metadata processing method further includes: obtaining the current service volume of the initial controller; inputting the service weight and the current service volume of the initial controller into the migration service volume calculation formula to obtain the migration service volume of the initial controller; wherein, the migration service volume of the initial controller represents the service volume that the initial controller needs to migrate to the newly added controller.

[0107] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: The migration service volume calculation formula is expressed as:

[0108]

[0109] Wherein, W outA Represents the migration service volume of the initial controller A, Score A Represents the service weight of the initial controller A, ΣCount curController Score represents the total service weight of multiple initial controllers; W A Represents the current service volume of the initial controller A.

[0110] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: determining the second service volume that matches the service weight of the non-removal controller, and processing the metadata to be removed of the to-be-removed controller based on the second service volume includes: determining the service weight of the non-removal controller based on the service weights of multiple initial controllers; calculating the service volume that the non-removal controller needs to bear based on the service weight of the non-removal controller, the current service volumes of multiple initial controllers, and the non-removal controller bearing service volume calculation formula; performing data migration on the metadata of multiple initial controllers based on the service volume that the non-removal controller needs to bear, so that the metadata of multiple initial controllers is evenly distributed on the non-removal controllers.

[0111] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: The non-removal controller bearing service volume calculation formula is expressed as:

[0112]

[0113] Wherein, W inB Represents the service volume that the non-removal controller B needs to bear, Score BIndicates the service weight of non-removable controller B, ΣCount avaController Score represents the total service weight of multiple non-removable controllers; W C Indicates the service volume to be removed.

[0114] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: Data migration includes: removing the metadata to be removed and the Internet protocol address corresponding to the metadata to be removed on the controller to be removed, and receiving the removed metadata and setting the Internet protocol address corresponding to the removed metadata on the receiving controller corresponding to the controller to be removed.

[0115] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: Before obtaining the controller operation type, it also includes: creating a file system, performing metadata service initialization based on the number of controllers in the current cluster, creating a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service statuses; in response to the completion of creating the file system view and the metadata service view, mounting the Internet protocol address corresponding to the controller to the file system, creating an Internet protocol address view, and the Internet protocol address map includes the Internet protocol addresses bound to the network interfaces of multiple controllers.

[0116] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: Before obtaining the controller operation type, it also includes: creating a file system, performing metadata service initialization based on the number of controllers in the current cluster, creating a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service statuses; in response to the completion of creating the file system view and the metadata service view, mounting the Internet protocol address corresponding to the controller to the file system, creating an Internet protocol address view, and the Internet protocol address map includes the Internet protocol addresses bound to the network interfaces of multiple controllers.

[0117] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: The method further includes: The metadata processing method further includes: in response to the controller operation type being a controller failure operation, selecting a non-faulty controller that meets the preset conditions from multiple initial controllers to take over the metadata of the faulty controller; wherein, the preset conditions include the service weights of other non-faulty controllers whose service weights are lower than the target quantity.

[0118] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: The metadata processing method further includes: in response to the controller operation type being a controller recovery operation type, determining a target recovery controller corresponding to the metadata service to be recovered, and binding the metadata service to be recovered with the target recovery controller based on the target recovery controller.

[0119] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: The metadata processing method further includes: obtaining the controller network interface status type, and determining the binding controller of the Internet protocol address corresponding to the network interface according to the controller network interface status type; wherein, when the controller network interface status type is a network interface failure, obtaining the order of the controller with the network interface failure in the chassis, determining the network interface of the binding controller according to the order of the controller in the chassis, and binding the network interface of the binding controller with the Internet protocol address corresponding to the faulty network interface.

[0120] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: Obtaining the controller network interface status type, and determining the binding controller of the Internet protocol address corresponding to the network interface according to the controller network interface status type further includes: when the controller network interface status type is a network interface recovery, the cluster status manager searches in the Internet protocol address view for the target controller corresponding to the Internet protocol address of the network interface to be recovered, the target controller is a specified running controller and the target controller meets that the Internet protocol address corresponding to the currently running network interface to be recovered does not belong to the target controller, and binding the Internet protocol address corresponding to the recovered network interface with the target controller.

[0121] In one embodiment, the above device can implement another implementation manner of the metadata processing method, and the specific steps are as follows: The metadata processing method further includes: obtaining the storage pool status type corresponding to the controller, and processing the metadata service corresponding to the controller according to the storage pool status type; wherein, when the storage pool status type corresponding to the controller is a storage pool offline, triggering a service interruption instruction to prohibit the operation of the metadata service corresponding to the offline storage pool.

[0122] In one embodiment, the above device can implement another implementation manner of the metadata processing method. The specific steps are as follows: Obtain the storage pool status type corresponding to the controller. Processing the metadata according to the storage pool status type further includes: When the storage pool status type corresponding to the controller is that the storage pool resumes online, trigger a service recovery start instruction, resume and start the metadata service on the controller corresponding to the file system on the recovered storage pool, trigger a metadata service status switching instruction, and switch the metadata service status to normal operation of the metadata service based on the metadata service status switching instruction.

[0123] As Figure 6 shown, an embodiment of the present application further provides a computer device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above embodiments of the metadata processing method.

[0124] An embodiment of the present application further provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. Wherein, the computer program is configured to execute the steps in any one of the above embodiments of the metadata processing method when running.

[0125] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), external hard drives, magnetic disks, or optical discs that can store computer programs.

[0126] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be determined to exceed the scope of the present application.

[0127] The above has introduced in detail a metadata processing method provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A metadata processing method, characterized in that: The method comprises: Get the controller action type; Inputting controller information of an initial controller into a business weight calculation formula to obtain a business weight of the initial controller, wherein the initial controller is each controller in the current cluster, and the controller information includes at least one of controller load information, controller failure rate, and controller delay information; In response to the controller operation type being a controller addition operation, the number of newly added controllers, the sum of the business weights of multiple initial controllers, and a first business volume calculation formula are obtained to calculate a first business volume; data migration is performed on newly added metadata and metadata of the initial controller according to the first business volume, and distribution of the newly added metadata and metadata of the initial controller to the initial controller and the newly added controller is balanced; the first business volume represents the business volume that the newly added controller needs to carry; In response to the controller operation type being a controller removal operation, a non-removal controller is selected from multiple initial controllers, and the business weight of the non-removal controller is determined based on the business weights of the multiple initial controllers; a second business volume is calculated based on the business weight of the non-removal controller, the current business volumes of the multiple initial controllers, and a calculation formula for the business volume carried by the non-removal controller; based on the second business volume, data migration is performed on the metadata of the multiple initial controllers so that the metadata of the multiple initial controllers are evenly distributed on the non-removal controller, and the second business volume represents the business volume that the non-removal controller needs to carry.

2. The metadata processing method according to claim 1, characterized in that: The business weight calculation formula is expressed as: Among them, Score A represents the service weight of the initial controller A, X represents the controller load impact weight, and CPU loadpct represents the load of the initial controller A, which is the ratio of the sum of the processes running and the processes waiting to be run of the initial controller A to the number of processor cores corresponding to the initial controller A. Y represents the impact weight of the controller's historical failure rate. HFR represents the failure rate of the initial controller A, which is the ratio of the number of failures of the initial controller A in the historical preset period to the preset period. Z represents the impact weight of the controller delay. latency It represents the average delay corresponding to the read and write operations of the initial controller A, H and M represent the adjustment parameters, and t represents the percentage.

3. The metadata processing method according to claim 1, characterized in that: The first business volume calculation formula is expressed as: Among them, W inM Indicates the amount of business that the newly added controller M needs to carry, Count newController Indicates the number of newly added controllers, Score i represents the score of the initial controller i, ΣCount curController Score represents the sum of the business weights of multiple initial controllers, W i Represents the current traffic volume of the initial controller i.

4. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: Get the current business volume of the initial controller; The service weight and current service volume of the initial controller are input into a migration service volume calculation formula to obtain the migration service volume of the initial controller; wherein the migration service volume of the initial controller represents the service volume that the initial controller needs to migrate to the newly added controller.

5. The metadata processing method according to claim 4, characterized in that: The calculation formula of the migration traffic volume is expressed as: Among them, W outA Indicates the migration business volume of the initial controller A, Score A Indicates the business weight of the initial controller A, ΣCount curController Score represents the sum of the business weights of multiple initial controllers; W A Indicates the current traffic volume of the initial controller A.

6. The metadata processing method according to claim 1, characterized in that: The calculation formula of the traffic volume carried by the non-removed controller is expressed as: Among them, W inB Indicates the business volume that the non-removed controller B needs to carry, Score B Indicates the business weight of the non-removed controller B, ΣCount avaController Score represents the sum of the business weights of multiple non-removed controllers; W C Indicates the business volume to be removed.

7. The metadata processing method according to claim 1, characterized in that: The data migration includes: The metadata to be removed and the Internet Protocol address corresponding to the metadata to be removed are removed on the controller to be removed, and the removed metadata is received and the Internet Protocol address corresponding to the removed metadata is set on the receiving controller corresponding to the controller to be removed.

8. The metadata processing method according to claim 7, characterized in that: The data migration also includes: In response to receiving the data migration instruction, obtaining a metadata service view and an Internet Protocol address view; Traversing the Internet Protocol address view, marking the Internet Protocol address corresponding to the metadata to be removed as the stop Internet Protocol address, and marking the Internet Protocol address corresponding to the metadata to be removed as the start Internet Protocol address; Traversing the metadata service view, marking the metadata to be removed as stop metadata, and marking the metadata to be removed as start metadata; Stop the operation of the Internet Protocol address running program of the marked stopped Internet Protocol address; in response to the successful stopping of the operation of the Internet Protocol address running program of the marked stopped Internet Protocol address, stop the operation of the metadata service corresponding to the marked stopped metadata; In response to the metadata service corresponding to the metadata marked to stop successfully stopping operation, starting the metadata service corresponding to the metadata marked to start; In response to the metadata service corresponding to the marked startup metadata being successfully started, the Internet Protocol address of the marked startup Internet Protocol address is started to run the program.

9. The metadata processing method according to claim 1, characterized in that: The method of obtaining the controller operation type also includes: Create a file system, perform metadata service initialization based on the number of controllers in the current cluster, and create a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service states; In response to the completion of the creation of the file system view and the metadata service view, the Internet Protocol address corresponding to the controller is mounted on the file system to create an Internet Protocol address view. The Internet Protocol address map includes Internet Protocol addresses bound to network ports of multiple controllers.

10. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: In response to the controller operation type being a controller failure operation, a non-faulty controller that meets preset conditions is selected from multiple initial controllers to take over metadata of the faulty controller; wherein the preset conditions include business weights of other non-faulty controllers whose business weights are lower than a target number.

11. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: In response to the controller operation type being a controller recovery operation type, a target recovery controller corresponding to the metadata service to be recovered is determined, and the metadata service to be recovered is bound to the target recovery controller based on the target recovery controller.

12. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: Obtain the controller network port status type, and determine the binding controller of the Internet Protocol address corresponding to the network port according to the controller network port status type; Among them, when the controller network port status type is network port failure, the order of the controllers with network port failure in the chassis is obtained, the network port of the bound controller is determined according to the order of the controllers in the chassis, and the network port of the bound controller is bound to the Internet Protocol address corresponding to the faulty network port.

13. The metadata processing method according to claim 12, characterized in that: The step of obtaining the controller network port status type and determining the binding controller of the Internet Protocol address corresponding to the network port according to the controller network port status type further includes: When the controller network port status type is network port recovery, the cluster status manager searches for the target controller corresponding to the Internet Protocol address corresponding to the network port to be restored from the Internet Protocol address view, the target controller is the designated running controller and the target controller satisfies the condition that the Internet Protocol address corresponding to the currently running network port to be restored does not belong to the target controller, and binds the Internet Protocol address corresponding to the recovery network port to the target controller.

14. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: Obtain the storage pool status type corresponding to the controller, and process the metadata service corresponding to the controller according to the storage pool status type; When the storage pool state type corresponding to the controller is that the storage pool is offline, a service interruption instruction is triggered to prohibit the running of the metadata service corresponding to the offline storage pool.

15. The metadata processing method according to claim 14, characterized in that: The obtaining of the storage pool status type corresponding to the controller and processing the metadata according to the storage pool status type further comprises: When the storage pool status type corresponding to the controller is that the storage pool is restored online, the business recovery startup instruction is triggered, and the metadata service on the controller corresponding to the file system on the restored storage pool is restored and started, triggering the metadata service status switching instruction, and based on the metadata service status switching instruction, the metadata service status is switched to normal operation of the metadata service.

16. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the metadata processing method according to any one of claims 1 to 15 when executing the computer program.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the metadata processing method according to any one of claims 1 to 15.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the metadata processing method according to any one of claims 1 to 15 are implemented.

Images