Metadata processing method, computer equipment, readable storage medium and program product

Abstract

The invention discloses a metadata processing method, computer equipment, a readable storage medium and a program product, and relates to the technical field of computers, the metadata processing method comprises the following steps: processing metadata based on different controller operation types and business weights of a plurality of initial controllers; the controller operation types comprise a controller newly-added operation type and a controller removing operation type, different metadata processing schemes are set for different controller operation types, rapid metadata migration under multiple scenes is supported, the technical problems that in the related technology, a metadata recovery service is single, and the recovery time is long are solved, and the method is suitable for being applied to the field of metadata migration. The technical effect of improving the reliability of the metadata service is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a metadata processing method, a computer device, a readable storage medium, and a program product. Background Art

[0002] When a metadata cluster fails, other metadata services will take over the failed metadata service and restore the cluster's external services. During the recovery period, the replacement metadata service does not provide external services, that is, it does not respond to client requests. Therefore, the length of the recovery time directly affects the user experience.

[0003] In the related technology, traditional file system metadata services are provided by a single controller, which has a single point of failure, and all IO (input / output) wait for the recovery of a single metadata service. Therefore, the existing metadata service cannot meet the requirements for metadata recovery time in fault drift scenarios, resulting in a decrease in metadata service reliability. Summary of the invention

[0004] The present application provides a metadata processing method, a computer device, a readable storage medium and a program product to solve the technical problem of a single metadata recovery service and a long recovery time in the related art.

[0005] The present application provides a metadata processing method, including: obtaining a controller operation type; obtaining a business weight of an initial controller; in response to the controller operation type being a controller addition operation, determining a first business volume that matches the business weight of the initial controller, and processing the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry; in response to the controller operation type being a controller removal operation, selecting a non-removed controller from multiple initial controllers, determining a second business volume that matches the business weight of the non-removed controller, and processing the metadata to be removed of the controller to be removed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

[0006] The present application also provides a computer device, including: a memory for storing a computer program; and a processor for implementing the steps of the metadata processing method in the following implementation mode when executing the computer program.

[0007] Obtain a controller operation type; obtain a business weight of an initial controller; in response to the controller operation type being a controller addition operation, determine a first business volume that matches the business weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry; in response to the controller operation type being a controller removal operation, select a non-removed controller from a plurality of initial controllers, determine a second business volume that matches the business weight of the non-removed controller, and process the metadata to be removed of the controller to be removed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata processing method in the following implementation are implemented.

[0009] Obtain a controller operation type; obtain a business weight of an initial controller; in response to the controller operation type being a controller addition operation, determine a first business volume that matches the business weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry; in response to the controller operation type being a controller removal operation, select a non-removed controller from a plurality of initial controllers, determine a second business volume that matches the business weight of the non-removed controller, and process the metadata to be removed of the controller to be removed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of the metadata processing method in the following implementation manner when the computer program is executed by a processor.

[0011] Obtain a controller operation type; obtain a business weight of an initial controller; in response to the controller operation type being a controller addition operation, determine a first business volume that matches the business weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry; in response to the controller operation type being a controller removal operation, select a non-removed controller from a plurality of initial controllers, determine a second business volume that matches the business weight of the non-removed controller, and process the metadata to be removed of the controller to be removed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

[0012] The metadata processing method provided in the present application processes metadata based on the controller operation type and the business weights of multiple initial controllers; wherein, the controller operation type is a controller addition operation, a first business volume matching the business weight of the initial controller is determined, and the newly added metadata and the metadata of the initial controller are processed based on the first business volume; wherein, the first business volume represents the business volume that the newly added controller needs to carry; the controller operation type is a controller removal operation, a non-removed controller is selected from multiple initial controllers, a second business volume matching the business weight of the non-removed controller is determined, and the metadata to be removed of the controller to be removed is processed based on the second business volume; wherein, the second business volume represents the business volume that the non-removed controller needs to carry. Therefore, it supports rapid metadata migration in multiple scenarios, solves the technical problems of a single metadata recovery service and a long recovery time in related technologies, and achieves the technical effect of improving the reliability of metadata services. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of the structure of a metadata processing system provided in one embodiment of the present application; Figure 2 A flowchart of a metadata processing method provided in one embodiment of the present application; Figure 3 A flowchart of a metadata processing method provided by another embodiment of the present application; Figure 4 A flowchart of a metadata processing method provided in yet another embodiment of the present application; Figure 5 A schematic diagram of the structure of a metadata processing device provided in one embodiment of the present application; Figure 6 This is a diagram of the internal structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0016] It should be noted that, in the description of this application, the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0017] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] like Figure 1 As shown, the metadata processing method of the present application is applied to a metadata processing system, which specifically includes: a host, multiple controllers (controller A and controller B shown in the figure), cluster management (cluster state manager) and a NAS pool.

[0019] The metadata processing system of the present application is implemented based on a centralized NAS architecture. NAS (Network Attached Storage) is a file system sharing device connected to a local area network based on an IP (Internet Protocol address). It separates storage devices and servers and allows users to access data through the network.

[0020] The host is the central component of the network with intelligence, responsible for running the software and hardware of all network devices and managing all data, files and information in the network.

[0021] The controller is an indispensable component of the automation system. It receives input signals from sensors or other devices and generates corresponding output signals through preset control strategies or algorithms, thereby achieving precise control of the controlled object.

[0022] As shown in the figure, in this application, metadata services are set on each controller, and each file system has a metadata service on each controller (a service that provides access to metadata, a system or platform for managing and operating metadata). The metadata services on all controllers form a cluster, and the failure of the metadata service on a controller does not affect the overall availability of the file system. The metadata service has 7 states: initial / loading / normal / unloading / unloaded / offline / deleted.

[0023] The controller also includes NAS network settings, NAS sharing protocol service settings, data space management settings, etc. The settings of NAS network, NAS sharing protocol service, and data space management are used to realize data sharing and file management of network attached storage devices.

[0024] Cluster management (Monitor) refers to the manager of cluster status, maintaining the status of the entire cluster and forming a cluster view together with other cluster status managers. Cluster managers can manage NAS file system views, metadata service views, and Internet protocol address views, use cluster interconnection links such as NTB / FC / RoCE for configuration synchronization and message transmission, and select the master controller through the PAXOS consistency algorithm.

[0025] NAS pool (Network Attached Storage) is a device used to store and share data. It is connected to the network in the local area network and provides file access services. NAS pool integrates the storage resources of multiple NAS devices into a unified storage pool to form a logically centralized storage system. After the storage resources are pooled, the administrator can manage and allocate storage resources through a unified interface without having to worry about specific physical devices. This can improve the utilization of storage resources, reduce management costs, and make the storage system more flexible and scalable.

[0026] like Figure 2 As shown, an embodiment of the present application provides a metadata processing method, which specifically includes the following steps: Step 101: Get the controller operation type.

[0027] Step 102: Obtain the service weight of the initial controller.

[0028] Specifically, the operation types of the controller include controller addition operations and controller removal operations. In the present application, a cluster including multiple controllers is set up. The controller addition operation refers to adding a new controller to the cluster, and the controller removal operation refers to removing a controller from the cluster. The initial controller refers to each controller in the current cluster.

[0029] The controller information may be obtained, and the controller information includes at least one of the controller load information, the controller failure rate, the controller delay information, etc. The controller information is input into the service weight calculation formula to obtain the service weight of the initial controller.

[0030] The controller load information may include the number of processor cores, which refers to the number of independent physical processing units inside the CPU, each of which can perform computing tasks; the number of processes the controller is running; and the number of processes the controller is waiting to run.

[0031] The controller failure rate may be the frequency at which the controller fails within a period of history, which may reflect the health status of the controller.

[0032] Controller latency information: Controller latency information may refer to the time required for data to travel from the controller to the controller receiving the data. Latency information is one of the core indicators for measuring network performance.

[0033] Specifically, the calculation formula of the business weight is as follows: ; Among them, Score A represents the service weight of the initial controller A, X represents the controller load impact weight, and CPU loadpct represents the load of the initial controller A, which is the ratio of the sum of the processes running and the processes waiting to be run of the initial controller A to the number of processor cores corresponding to the initial controller A. Y represents the impact weight of the controller's historical failure rate. HFR represents the failure rate of the initial controller A, which is the ratio of the number of failures of the initial controller A in the historical preset period to the preset period. Z represents the impact weight of the controller delay. latency It represents the average delay corresponding to the read and write operations of the initial controller A, H and M represent the adjustment parameters, and t represents the percentage.

[0034] For example, if the number of CPU cores of controller A is 40, and the number of running processes and waiting processes is 10, then the initial load of controller A is 0.25; assuming that controller A fails twice within 100 days, the controller failure rate is 0.02.

[0035] The business weight of a controller reflects the business carrying capacity of the controller. The lower the business weight of a controller, the less business the controller should carry. Conversely, the higher the business weight of a controller, the higher the business volume the controller should carry.

[0036] By comprehensively considering the controller load, controller failure rate, and controller latency (average IO latency), the service weight of each controller is determined and then used to calculate the service volume carried by the controller, thereby maximizing the load balance of the entire cluster.

[0037] The operation types of this application provide different metadata processing solutions for different controller operation types.

[0038] Step 1021: In response to the controller operation type being a controller addition operation, determine a first business volume that matches the business weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry.

[0039] Specifically, the number of new controllers, the sum of the business weights of multiple initial controllers and the first business volume calculation formula are obtained to calculate the business volume that the new controllers need to carry; data migration is performed on the new metadata and the metadata of the initial controllers according to the business volume that the new controllers need to carry, and the distribution of the new metadata and the metadata of the initial controllers in the initial controllers and the new controllers is balanced.

[0040] The first business volume calculation formula may be as follows: ; Among them, W inM Indicates the amount of business that the newly added controller M needs to carry, Count newController Indicates the number of newly added controllers, Score i represents the score of the initial controller i, ΣCount curController Score represents the sum of the business weights of multiple initial controllers, W i Represents the current traffic volume of the initial controller i.

[0041] In one embodiment, the metadata processing method also includes: obtaining the current business volume of the initial controller; inputting the business weight and current business volume of the initial controller into the migration business volume calculation formula to obtain the migration business volume of the initial controller; wherein the migration business volume of the initial controller represents the business volume that the initial controller needs to migrate to the newly added controller.

[0042] The calculation formula of the migration business volume is expressed as: ; Among them, W outA Indicates the migration business volume of the initial controller A, Score A Indicates the business weight of the initial controller A, ΣCount curController Score represents the sum of the business weights of multiple initial controllers; W A Indicates the current traffic volume of the initial controller A.

[0043] When a controller is added, the cluster state manager will initialize a new metadata service, that is, configure the metadata service on the newly added controller, and bind the corresponding metadata service to the newly added controller according to the controller location information of the newly added controller. At this time, the cluster state manager will trigger dynamic load balancing, and calculate the business volume that needs to be carried on the newly added controller in the scenario of the newly added controller through the first business volume calculation formula and the migration business volume calculation formula, and the business volume that each initial controller in the cluster needs to migrate to the newly added controller. In this way, after a new controller is added to the cluster, load balancing can be achieved between the newly added controller and the initial controller in the cluster.

[0044] Step 1022: In response to the controller operation type being a controller removal operation, a non-removed controller is selected from multiple initial controllers, a second business volume that matches the business weight of the non-removed controller is determined, and the metadata to be removed of the controller to be removed is processed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

[0045] Specifically, the business weight of the non-removed controller is determined based on the business weights of multiple initial controllers; the business volume that the non-removed controller needs to carry is calculated based on the business weight of the non-removed controller, the current business volume of the multiple initial controllers and the business volume calculation formula of the non-removed controller; based on the business volume that the non-removed controller needs to carry, the metadata of the multiple initial controllers are migrated so that the metadata of the multiple initial controllers are evenly distributed on the non-removed controller.

[0046] Specifically, the calculation formula for the traffic volume carried by the non-removed controller is expressed as: ; Among them, W inB Indicates the business volume that the non-removed controller B needs to carry, Score B Indicates the business weight of the non-removed controller B, ΣCount avaController Score represents the sum of the business weights of multiple non-removed controllers; W C Indicates the business volume to be removed.

[0047] A non-removable controller refers to a controller that is not removed from the cluster. When a controller is removed from the cluster, load balancing will be triggered. According to the business weight of each non-removable controller, the metadata carried by the removed controller needs to be distributed to the non-removable controllers in the cluster to ensure load balancing between the non-removable controllers that have taken over the metadata on the removed controller.

[0048] In this application, according to the different scene changes of the controller and the calculation formulas corresponding to different scenes, both the metadata processing efficiency of scene switching and the security of data can be considered.

[0049] In one embodiment, before obtaining the controller operation type, it also includes: creating a file system, performing metadata service initialization based on the number of controllers in the current cluster, adding metadata services for each controller in the cluster, creating a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service states; in response to the completion of the creation of the file system view and the metadata service view, the host sends a read and write request, mounts the file system with the Internet Protocol address corresponding to the controller, creates an Internet Protocol address view, and the Internet Protocol address map includes the Internet Protocol addresses bound to the network ports of multiple controllers, so that the controller responds to the read and write request sent by the host to access the file system mounted by the controller. By creating the file system view, the metadata service view, and the Internet Protocol address view, the metadata service state, the Internet Protocol address binding information, etc. can be obtained in a timely manner. By adopting the multi-data service, the cluster state manager maintains the metadata service view and controls the start and stop of the metadata service.

[0050] In actual applications, users perform file system configuration through GUI (Graphical User Interface) / CLI (Command Line Interface). Mon (Cluster State Manager) selects to initialize MDS (Metadata Service) according to the number of controllers in the current cluster. According to the location information of N controllers, the MDS after initialization is divided into instance0~instance(N-1). In normal state, instanceX (instance X) is on the X controller in the cluster. Since MDS is the implementation of the file system, it does not need to be specified by the user and is bound to the location information of the controller. Therefore, the metadata service only has the current running controller attribute, and the controller to which it belongs is calculated in real time through the location information. In other words, the user only cares about the created file system, but this file system is still carried by several metadata services. It is an internal implementation and is not displayed to the outside. As long as its instance number is fixed internally, for example, instance is on the first controller, instancen-1. The metadata service state transition during the configuration process involves initial->loading->normal.

[0051] The user performs IP configuration through the GUI / CLI. The parameters include the specified controller and network port, that is, the preferred controller and network port information of the IP (Internet Protocol address) in the normal state. The cluster state manager adds the IP to the specified port on the specified controller through the two parameters of controller and network port, and then informs all controllers through broadcast to update the Internet Protocol address view. The IP state transition during the configuration process includes initial->loading->normal. When the network port fails or the controller fails, the IP configured on the network port is configured by the cluster state manager on another controller to achieve service availability in the failure scenario. The IP contains two attributes: the preferred controller and the bound controller, that is, the running controller specified during configuration and the currently running controller. The IP has five states: initial / loading / normal / unloading / unloaded. Of course, the cluster state manager will also notify all controllers in the MDS cluster of the updated metadata service view.

[0052] When the controller operation type is a controller adding operation type and when the controller operation type is a controller removing operation type, data migration is involved. Here, data migration includes removing the metadata to be removed and the Internet Protocol address corresponding to the metadata to be removed on the controller to be removed, receiving the removed metadata on the receiving controller corresponding to the controller to be removed, and setting the Internet Protocol address corresponding to the removed metadata. That is, when performing metadata migration, the metadata and its corresponding Internet Protocol address need to be unbound from the controller to be removed and rebound to the receiving controller.

[0053] The data migration specifically includes: in response to receiving a data migration instruction, obtaining a metadata service view and an Internet Protocol address view; traversing the Internet Protocol address view, marking the Internet Protocol address corresponding to the metadata to be removed as a stop Internet Protocol address, and marking the Internet Protocol address corresponding to the received removal metadata as a start Internet Protocol address; traversing the metadata service view, marking the metadata to be removed as stop metadata, and marking the received removal metadata as start metadata; stopping the Internet Protocol address running program marked with the stop Internet Protocol address; in response to successfully stopping the Internet Protocol address running program marked with the stop Internet Protocol address, stopping the metadata service corresponding to the marked stop metadata; in response to successfully stopping the metadata service corresponding to the marked stop metadata, starting the metadata service corresponding to the marked start metadata; in response to successfully starting the metadata service corresponding to the marked start metadata, starting the Internet Protocol address running program marked with the start Internet Protocol address.

[0054] like Figure 3 As shown, specifically, stop running the corresponding unbinding operation, start the corresponding binding operation, traverse the Internet Protocol address view, mark the IP that needs to be started and the IP that needs to be stopped, traverse the metadata service view, mark the MDS (metadata service) that needs to be started and the MDS that needs to be stopped, traverse all IPs that need to be stopped, and perform a stop action on the corresponding currently running controller until all IPs that need to be stopped have been stopped, the stop action can be concurrent, and continue after completion, otherwise wait; traverse all MDSs that need to be stopped, and perform a stop action on the corresponding currently running controller until all MDSs that need to be stopped have been stopped, the stop action can be concurrent, and continue after completion, otherwise wait; traverse all MDSs that need to be started, and perform a start action on the controller that needs to be started until all MDSs that need to be started are started, the start action can be concurrent, and continue after completion, otherwise wait; traverse all IPs that need to be started, and perform a start action on the controller that needs to be started until all IPs that need to be started are started, the start action can be concurrent, and continue after completion, otherwise wait; That is, you need to start the metadata service first, then configure the IP, prepare the metadata service first, and then provide IP access to the outside world. If the IP is configured first and the metadata service is not configured yet, an error will be reported.

[0055] In one embodiment, the controller operation type also includes a controller failure operation type, and the metadata processing method also includes: in response to the controller operation type being a controller failure operation, selecting a non-faulty controller that meets preset conditions from multiple initial controllers to take over the metadata of the faulty controller; wherein the preset conditions include the business weights of other non-faulty controllers whose business weights are lower than the target number.

[0056] The non-faulty controller with the lowest business weight or the target number of non-faulty controllers with low business weight can be directly selected to take over the metadata of the faulty controller. The selection of the number of non-faulty controllers here depends on the business volume of the metadata of the faulty controller and the business acceptance capacity of the non-faulty controller with low business weight.

[0057] Controller failure is different from controller removal. Controller removal refers to a long-term failure, and the unusable controller will be removed, while controller failure refers to a short-term failure, such as controller disconnection, controller power outage, etc. The controller is unavailable for a short period of time. In this application, the old controller failure will not remove the metadata service of the failed controller. The controller with the smallest business weight will be selected to completely take over the business of the failed controller. The disadvantage is that the business pressure is uneven for a short period of time, but considering that the controller failure is not a long-term stable state, it is acceptable. The advantage of this is to increase the speed of business migration and avoid splitting the business.

[0058] In one embodiment, the controller operation type also includes controller recovery, and the metadata processing method also includes: in response to the controller operation type being a controller recovery operation type, determining a target recovery controller corresponding to the metadata service to be restored, and binding the metadata service to be restored to the target recovery controller based on the target recovery controller.

[0059] Controller recovery is the reverse operation of controller failure. The cluster state manager will use the preferred binding controller of the metadata service to be recovered as the target recovery controller, bind the metadata service to be recovered with the target recovery controller, and broadcast the updated metadata service view to all controllers. Each controller determines whether to start or stop the corresponding metadata service based on the current running metadata service status and the latest view.

[0060] In one embodiment, the metadata processing method further includes: obtaining the controller network port status type, and determining the bound controller of the Internet Protocol address corresponding to the network port according to the controller network port status type; wherein, when the controller network port status type is network port failure, obtaining the order of the controller with network port failure in the chassis, determining the network port of the bound controller according to the order of the controllers in the chassis, and binding the network port of the bound controller to the Internet Protocol address corresponding to the failed network port. When the controller network port status type is network port recovery, the cluster state manager searches the Internet Protocol address view for the target controller corresponding to the Internet Protocol address corresponding to the network port to be recovered, and the target controller is the designated running controller and the target controller satisfies the Internet Protocol address corresponding to the current running network port to be recovered does not belong to the target controller, and binds the Internet Protocol address corresponding to the recovery network port to the target controller.

[0061] After a network port fails, you need to select another network port to configure the Internet Protocol address. By default, the metadata service searches for ports in the same position as the network ports of the bound controllers according to the order of the controllers in the chassis, and then broadcasts the Internet Protocol address view to all controllers. Each controller chooses to stop or start the IP based on the bound controller information of the IP in the view. That is, if the bound controller is not the current controller, but the IP is running on the current controller, then stop the IP; if the bound controller is the current controller, but the IP is not running on the current controller, then start the IP. In other words, the target controller is the designated running controller and the target controller meets the Internet Protocol address corresponding to the current running network port to be restored that does not belong to the target controller.

[0062] like Figure 4As shown, in order to avoid data inconsistency, the switching order needs to be ensured as follows: first, the cluster status manager will stop the running program of the IP with the network port fault, and return to the cluster status manager that the running program of the IP with the network port fault has stopped successfully. In response to the cluster status manager receiving the running program of the IP with the network port fault has stopped successfully, the running program of the IP of the target controller is started, and the running program of the IP of the target controller is returned to the cluster status manager that the running program of the IP has started successfully.

[0063] After the network port of a controller is restored, Mon traverses the complete Internet Protocol address view, searches for the IP of the preferred controller that belongs to the controller and the currently running controller that does not belong to the controller, updates the bound controller in the Internet Protocol address view to the controller, and broadcasts the Internet Protocol address view. Each controller chooses to stop or start the IP based on the bound controller information in the view.

[0064] In one embodiment, the present application further includes obtaining the storage pool status type corresponding to the controller, and processing the metadata service corresponding to the controller according to the storage pool status type; wherein, when the storage pool status type corresponding to the controller is that the storage pool is offline, a service interruption instruction is triggered to prohibit the operation of the metadata service corresponding to the offline storage pool. When the storage pool status type corresponding to the controller is that the storage pool is restored online, a service recovery start instruction is triggered to restore and start the metadata service on the controller corresponding to the file system on the restored storage pool, and a metadata service status switching instruction is triggered to switch the metadata service status to normal operation of the metadata service based on the metadata service status switching instruction.

[0065] After the storage pool is offline, the metadata service on the storage pool cannot be started normally no matter which controller it is migrated to. At this time, the business is interrupted and the metadata service does not need to be migrated. In other words, the metadata on the controller corresponding to the metadata service is not migrated, and the metadata service status switches to normal -> unloading -> unloaded -> offline.

[0066] After the storage pool is restored and online, the MDS cluster corresponding to the file system on the storage pool needs to be restored and started, triggering the state machine switch. The MDS state switches to offline -> loading -> normal. By starting the metadata service instance on each controller to form a metadata service cluster, the single point failure problem of the metadata service is solved. By controlling the drift order of the metadata service (MDS) and IP (Internet Protocol address), the efficiency and security of business drift in scenarios such as failures are solved.

[0067] In a specific implementation, the metadata processing method of the present application may be shown in the following steps: S1: When the service request is a metadata write request, the size of the metadata to be written to the controller and the size of the remaining data storage space corresponding to the cluster are obtained; S2: Compare the size of the remaining data storage space corresponding to the cluster with the size of the metadata to be written into the controller to obtain a comparison result.

[0068] S21: When the size of the remaining data storage space corresponding to the cluster is greater than or equal to the size of the metadata to be written to the controller, it is considered that no new controller is needed, the initial number of controllers in the cluster is obtained, the initial number of controllers in the cluster is used as the divisor, the size of the metadata to be written to the controller is used as the dividend, and the average value of the metadata to be written to each controller is calculated; S212: Obtain the remaining data storage space size of each initial controller in the cluster, compare the average value of the metadata to be written of each controller with the remaining data storage space size of the controller corresponding to the initial controller, and obtain a comparison result.

[0069] S2121: If the remaining data storage space size of any initial controller is smaller than the average value of the metadata to be written of each controller, the allocation ratio is determined based on the remaining data storage space size of each initial controller, and based on the allocation ratio, the metadata to be written is allocated to each initial controller in the cluster.

[0070] Here, the remaining data storage space size of the initial controller that is smaller than the average of the metadata to be written of each controller can be used as the first allocated storage space size, select the target metadata to be written that is smaller than or equal to the first allocated storage space size and allocate it to the controller corresponding to the first allocated storage space size, calculate the difference between the metadata to be written and the target metadata to be written, use this difference as the metadata to be allocated, obtain the initial controllers in the cluster that have not allocated metadata to be written and their corresponding business weights, allocate a higher allocation ratio to the initial controllers that have not allocated metadata to be written and have higher business weights, and allocate the metadata to be allocated to the initial controllers that have not allocated metadata to be written based on the allocation ratio. In this way, the remaining data storage space size on each initial controller can be fully considered, so that the metadata to be written is evenly distributed on each initial controller.

[0071] S2122: If the remaining data storage space of each initial controller is larger than the average value of the metadata to be written in each controller, the metadata to be written is evenly distributed to each controller according to the average value of the metadata to be written in each controller.

[0072] S22: When the remaining data storage space corresponding to the cluster is smaller than the size of the metadata to be written to the controller, it is considered that a new controller needs to be added, and the cluster state manager (Mon) triggers the addition of dynamic load balancing.

[0073] The number of newly added controllers is determined based on the size of metadata to be written to the controller and the size of the remaining data storage space corresponding to the cluster.

[0074] S22: In response to triggering the newly added dynamic load balancing, controller load information, controller failure rate and controller delay information are obtained, and the business acceptance weight of each initial controller is determined based on the controller load information, controller failure rate and controller delay information.

[0075] S23: Calculate the first service volume based on the service weight of each initial controller.

[0076] S24: Process the newly added metadata and the metadata of the initial controller based on the first business volume.

[0077] Before obtaining the size of metadata to be written to the controller and the size of the remaining data storage space corresponding to the cluster, it is also possible to determine whether the controller has a long-term fault, the storage pool status, the controller network port status, etc., and perform different metadata processing operations according to different status.

[0078] The embodiment of the present application provides a metadata processing device, the metadata processing device is specifically as follows Figure 5 As shown, the metadata processing device includes: an acquisition module 20 and a processing module 21.

[0079] An acquisition module 20 is used to acquire a controller operation type and acquire a service weight of an initial controller; Processing module 21 is used to, in response to the controller operation type being a controller addition operation, determine a first business volume that matches the business weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry; in response to the controller operation type being a controller removal operation, select a non-removed controller from multiple initial controllers, determine a second business volume that matches the business weight of the non-removed controller, and process the metadata to be removed of the controller to be removed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

[0080] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: obtaining the business weight of the initial controller includes: obtaining controller information of the initial controller; wherein the controller information includes at least one of controller load information, controller failure rate and controller delay information; inputting the controller information into the business weight calculation formula to obtain the business weight of the initial controller.

[0081] In one embodiment, the above device can implement another implementation method of the metadata processing method, and the specific steps are as follows: The service weight calculation formula is expressed as: ; Among them, Score A represents the service weight of the initial controller A, X represents the controller load impact weight, and CPU loadpct represents the load of the initial controller A, which is the ratio of the sum of the processes running and the processes waiting to be run of the initial controller A to the number of processor cores corresponding to the initial controller A. Y represents the impact weight of the controller's historical failure rate. HFR represents the failure rate of the initial controller A, which is the ratio of the number of failures of the initial controller A in the historical preset period to the preset period. Z represents the impact weight of the controller delay. latency It represents the average delay corresponding to the read and write operations of the initial controller A, H and M represent the adjustment parameters, and t represents the percentage.

[0082] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: determine a first business volume that matches the business weight of the initial controller, and process the newly added metadata and the metadata of the initial controller based on the first business volume, including: obtaining the number of newly added controllers, the sum of the business weights of multiple initial controllers, and the first business volume calculation formula to calculate the business volume that the newly added controller needs to carry; perform data migration on the newly added metadata and the metadata of the initial controller according to the business volume that the newly added controller needs to carry, and balance the distribution of the newly added metadata and the metadata of the initial controller in the initial controller and the newly added controller.

[0083] In one embodiment, the above device can implement another implementation of the metadata processing method, and the specific steps are as follows: The first business volume calculation formula is expressed as: ; Among them, W inM Indicates the amount of business that the newly added controller M needs to carry, Count newController Indicates the number of newly added controllers, Score i represents the score of the initial controller i, ΣCount curController Score represents the sum of the business weights of multiple initial controllers, W i Represents the current traffic volume of the initial controller i.

[0084] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: the metadata processing method also includes: obtaining the current business volume of the initial controller; inputting the business weight and current business volume of the initial controller into the migration business volume calculation formula to obtain the migration business volume of the initial controller; wherein the migration business volume of the initial controller represents the business volume that the initial controller needs to migrate to the newly added controller.

[0085] In one embodiment, the above device can implement another implementation of the metadata processing method, and the specific steps are as follows: The migration business volume calculation formula is expressed as: ; Among them, W outA Indicates the migration business volume of the initial controller A, Score A Indicates the business weight of the initial controller A, ΣCount curController Score represents the sum of the business weights of multiple initial controllers; W A Indicates the current traffic volume of the initial controller A.

[0086] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: determine a second business volume that matches the business weight of the non-removed controller, and process the metadata to be removed of the controller to be removed based on the second business volume, including: determining the business weight of the non-removed controller based on the business weights of multiple initial controllers; calculating the business volume that the non-removed controller needs to carry based on the business weight of the non-removed controller, the current business volume of multiple initial controllers and the non-removed controller carrying business volume calculation formula; based on the business volume that the non-removed controller needs to carry, perform data migration on the metadata of multiple initial controllers, so that the metadata of multiple initial controllers are evenly distributed on the non-removed controller.

[0087] In one embodiment, the above device can implement another implementation of the metadata processing method, and the specific steps are as follows: The calculation formula of the traffic carried by the non-removed controller is expressed as: ; Among them, W inB Indicates the business volume that the non-removed controller B needs to carry, Score B Indicates the business weight of the non-removed controller B, ΣCount avaController Score represents the sum of the business weights of multiple non-removed controllers; W C Indicates the business volume to be removed.

[0088] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: data migration includes: removing the metadata to be removed and the Internet Protocol address corresponding to the metadata to be removed on the controller to be removed, receiving the removed metadata and setting the Internet Protocol address corresponding to the removed metadata on the receiving controller corresponding to the controller to be removed.

[0089] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: data migration includes: in response to receiving a data migration instruction, obtaining a metadata service view and an Internet Protocol address view; traversing the Internet Protocol address view, marking the Internet Protocol address corresponding to the metadata to be removed as a stop Internet Protocol address, and marking the Internet Protocol address corresponding to the received removal metadata as a start Internet Protocol address; traversing the metadata service view, marking the metadata to be removed as stop metadata, and marking the received removal metadata as start metadata; stopping the Internet Protocol address running program marked with the stop Internet Protocol address; in response to the Internet Protocol address running program marked with the stop Internet Protocol address stopping successfully, stopping the metadata service corresponding to the metadata marked with the stop metadata; in response to the metadata service corresponding to the metadata marked with the stop metadata stopping successfully, starting the metadata service corresponding to the metadata marked with the start metadata; in response to the metadata service corresponding to the metadata marked with the start metadata executing successfully, starting the Internet Protocol address running program marked with the start Internet Protocol address.

[0090] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: before obtaining the controller operation type, it also includes: creating a file system, performing metadata service initialization based on the number of controllers in the current cluster, and creating a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service states; in response to the completion of the creation of the file system view and the metadata service view, mounting the Internet Protocol address corresponding to the controller on the file system, creating an Internet Protocol address view, and the Internet Protocol address map includes the Internet Protocol addresses bound to the network ports of multiple controllers.

[0091] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: the method also includes: the metadata processing method also includes: in response to the controller operation type being a controller failure operation, selecting a non-faulty controller that meets preset conditions from multiple initial controllers to take over the metadata of the faulty controller; wherein the preset conditions include the business weights of other non-faulty controllers whose business weights are lower than the target number.

[0092] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: the metadata processing method also includes: in response to the controller operation type being a controller recovery operation type, determining the target recovery controller corresponding to the metadata service to be restored, and binding the metadata service to be restored to the target recovery controller based on the target recovery controller.

[0093] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: the metadata processing method also includes: obtaining the controller network port status type, and determining the binding controller of the Internet Protocol address corresponding to the network port according to the controller network port status type; wherein, when the controller network port status type is a network port failure, obtaining the order of the controllers with network port failure in the chassis, determining the network port of the binding controller according to the order of the controllers in the chassis, and binding the network port of the bound controller to the Internet Protocol address corresponding to the faulty network port.

[0094] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: obtaining the controller network port status type, determining the binding controller of the Internet Protocol address corresponding to the network port according to the controller network port status type, and further comprising: when the controller network port status type is network port recovery, the cluster status manager searches for the target controller corresponding to the Internet Protocol address corresponding to the network port to be restored from the Internet Protocol address view, the target controller is the designated running controller and the target controller satisfies that the Internet Protocol address corresponding to the currently running network port to be restored does not belong to the target controller, and binds the Internet Protocol address corresponding to the recovery network port to the target controller.

[0095] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: the metadata processing method also includes: obtaining the storage pool status type corresponding to the controller, and processing the metadata service corresponding to the controller according to the storage pool status type; wherein, when the storage pool status type corresponding to the controller is that the storage pool is offline, a business interruption instruction is triggered to prohibit the running of the metadata service corresponding to the offline storage pool.

[0096] In one embodiment, the above-mentioned device can implement another implementation method of the metadata processing method, and the specific steps are as follows: obtaining the storage pool status type corresponding to the controller, and processing the metadata according to the storage pool status type also includes: when the storage pool status type corresponding to the controller is that the storage pool is restored online, triggering a business recovery start instruction, restoring and starting the metadata service on the controller corresponding to the file system on the restored storage pool, triggering a metadata service status switching instruction, and switching the metadata service status to normal operation of the metadata service based on the metadata service status switching instruction.

[0097] like Figure 6As shown, an embodiment of the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above metadata processing method embodiments.

[0098] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above-mentioned metadata processing method embodiments when running.

[0099] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0100] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be judged to be beyond the scope of this application.

[0101] The above is a detailed introduction to a metadata processing method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A metadata processing method, characterized in that: The method comprises: Get the controller action type; Get the business weight of the initial controller; In response to the controller operation type being a controller addition operation, determining a first business volume that matches the business weight of the initial controller, and processing the newly added metadata and the metadata of the initial controller based on the first business volume; wherein the first business volume represents the business volume that the newly added controller needs to carry; In response to the controller operation type being a controller removal operation, a non-removed controller is selected from multiple initial controllers, a second business volume that matches the business weight of the non-removed controller is determined, and the metadata to be removed of the controller to be removed is processed based on the second business volume; wherein the second business volume represents the business volume that the non-removed controller needs to carry.

2. The metadata processing method according to claim 1, characterized in that: The obtaining of the service weight of the initial controller includes: Acquire controller information of the initial controller; wherein the controller information includes at least one of controller load information, controller failure rate, and controller delay information; The controller information is input into a business weight calculation formula to obtain the business weight of the initial controller.

3. The metadata processing method according to claim 2, characterized in that: The business weight calculation formula is expressed as: ； Among them, Score A represents the service weight of the initial controller A, X represents the controller load impact weight, and CPU loadpct represents the load of the initial controller A, which is the ratio of the sum of the processes running and the processes waiting to be run of the initial controller A to the number of processor cores corresponding to the initial controller A. Y represents the impact weight of the controller's historical failure rate. HFR represents the failure rate of the initial controller A, which is the ratio of the number of failures of the initial controller A in the historical preset period to the preset period. Z represents the impact weight of the controller delay. latency It represents the average delay corresponding to the read and write operations of the initial controller A, H and M represent the adjustment parameters, and t represents the percentage.

4. The metadata processing method according to claim 1, characterized in that: The determining of a first business volume matching the business weight of the initial controller, and processing the newly added metadata and the metadata of the initial controller based on the first business volume includes: Obtain the number of newly added controllers, the sum of the service weights of multiple initial controllers, and a first service volume calculation formula to calculate the service volume that the newly added controller needs to carry; Data migration is performed on the newly added metadata and the metadata of the initial controller according to the business volume that the newly added controller needs to carry, and the distribution of the newly added metadata and the metadata of the initial controller to the initial controller and the newly added controller is balanced.

5. The metadata processing method according to claim 4, characterized in that: The first business volume calculation formula is expressed as: ； Among them, W inM Indicates the amount of business that the newly added controller M needs to carry, Count newController Indicates the number of newly added controllers, Score i represents the score of the initial controller i, ΣCount curController Score represents the sum of the business weights of multiple initial controllers, W i Represents the current traffic volume of the initial controller i.

6. The metadata processing method according to claim 4, characterized in that: The metadata processing method further includes: Get the current business volume of the initial controller; The service weight and current service volume of the initial controller are input into a migration service volume calculation formula to obtain the migration service volume of the initial controller; wherein the migration service volume of the initial controller represents the service volume that the initial controller needs to migrate to the newly added controller.

7. The metadata processing method according to claim 6, characterized in that: The calculation formula of the migration traffic volume is expressed as: ; Among them, W outA Indicates the migration business volume of the initial controller A, Score A Indicates the business weight of the initial controller A, ΣCount curController Score represents the sum of the business weights of multiple initial controllers; W A Indicates the current traffic volume of the initial controller A.

8. The metadata processing method according to claim 1, characterized in that: The determining of a second service volume that matches the service weight of the controller not to be removed, and processing metadata to be removed of the controller to be removed based on the second service volume comprises: Determining a service weight of a non-removed controller based on the service weights of a plurality of initial controllers; Calculate the traffic volume that the non-removal controller needs to carry based on the traffic weight of the non-removal controller, the current traffic volume of the multiple initial controllers, and a calculation formula for the traffic volume carried by the non-removal controller; The metadata of the multiple initial controllers are migrated based on the traffic volume that the non-removed controller needs to bear, so that the metadata of the multiple initial controllers are evenly distributed on the non-removed controllers.

9. The metadata processing method according to claim 8, characterized in that: The calculation formula of the traffic volume carried by the non-removed controller is expressed as: ； Among them, W inB Indicates the business volume that the non-removed controller B needs to carry, Score B Indicates the business weight of the non-removed controller B, ΣCount avaController Score represents the sum of the business weights of multiple non-removed controllers; W C Indicates the business volume to be removed.

10. The metadata processing method according to claim 4 or 8, characterized in that: The data migration includes: The metadata to be removed and the Internet Protocol address corresponding to the metadata to be removed are removed on the controller to be removed, and the removed metadata is received and the Internet Protocol address corresponding to the removed metadata is set on the receiving controller corresponding to the controller to be removed.

11. The metadata processing method according to claim 10, characterized in that: The data migration includes: In response to receiving the data migration instruction, obtaining a metadata service view and an Internet Protocol address view; Traversing the Internet Protocol address view, marking the Internet Protocol address corresponding to the metadata to be removed as the stop Internet Protocol address, and marking the Internet Protocol address corresponding to the metadata to be removed as the start Internet Protocol address; Traversing the metadata service view, marking the metadata to be removed as stop metadata, and marking the metadata to be removed as start metadata; Causing a program running an Internet Protocol address that marks the Internet Protocol address as inactive to cease operation; In response to the Internet Protocol address running program of the marked stop Internet Protocol address being stopped successfully, the metadata service corresponding to the marked stop metadata is stopped; In response to the metadata service corresponding to the metadata marked to stop successfully stopping operation, starting the metadata service corresponding to the metadata marked to start; In response to the metadata service corresponding to the marked startup metadata being successfully started, the Internet Protocol address running program of the marked startup Internet Protocol address is started.

12. The metadata processing method according to claim 1, characterized in that: The method of obtaining the controller operation type also includes: Create a file system, perform metadata service initialization based on the number of controllers in the current cluster, and create a file system view and a metadata service view; the file system view includes the file system and its corresponding multiple metadata services; the metadata service view includes multiple controllers and their corresponding metadata service states; In response to the completion of the creation of the file system view and the metadata service view, the Internet Protocol address corresponding to the controller is mounted on the file system to create an Internet Protocol address view. The Internet Protocol address map includes Internet Protocol addresses bound to network ports of multiple controllers.

13. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: In response to the controller operation type being a controller failure operation, a non-faulty controller that meets preset conditions is selected from multiple initial controllers to take over metadata of the faulty controller; wherein the preset conditions include business weights of other non-faulty controllers whose business weights are lower than a target number.

14. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: In response to the controller operation type being a controller recovery operation type, a target recovery controller corresponding to the metadata service to be recovered is determined, and the metadata service to be recovered is bound to the target recovery controller based on the target recovery controller.

15. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: Obtain the controller network port status type, and determine the binding controller of the Internet Protocol address corresponding to the network port according to the controller network port status type; Among them, when the controller network port status type is network port failure, the order of the controllers with network port failure in the chassis is obtained, the network port of the bound controller is determined according to the order of the controllers in the chassis, and the network port of the bound controller is bound to the Internet Protocol address corresponding to the faulty network port.

16. The metadata processing method according to claim 15, characterized in that: The step of obtaining the controller network port status type and determining the binding controller of the Internet Protocol address corresponding to the network port according to the controller network port status type further includes: When the controller network port status type is network port recovery, the cluster status manager searches for the target controller corresponding to the Internet Protocol address corresponding to the network port to be restored from the Internet Protocol address view, the target controller is the designated running controller and the target controller satisfies the condition that the Internet Protocol address corresponding to the currently running network port to be restored does not belong to the target controller, and binds the Internet Protocol address corresponding to the recovery network port to the target controller.

17. The metadata processing method according to claim 1, characterized in that: The metadata processing method further includes: Obtain the storage pool status type corresponding to the controller, and process the metadata service corresponding to the controller according to the storage pool status type; When the storage pool state type corresponding to the controller is that the storage pool is offline, a service interruption instruction is triggered to prohibit the running of the metadata service corresponding to the offline storage pool.

18. The metadata processing method according to claim 17, characterized in that: The obtaining of the storage pool status type corresponding to the controller and processing the metadata according to the storage pool status type further comprises: When the storage pool status type corresponding to the controller is that the storage pool is restored online, the business recovery startup instruction is triggered, and the metadata service on the controller corresponding to the file system on the restored storage pool is restored and started, triggering the metadata service status switching instruction, and based on the metadata service status switching instruction, the metadata service status is switched to normal operation of the metadata service.

19. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the metadata processing method according to any one of claims 1 to 18 when executing the computer program.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the metadata processing method according to any one of claims 1 to 18.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the metadata processing method according to any one of claims 1 to 18 are implemented.

Images