Method and apparatus for accessing distributed storage cluster, device, and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for accessing a distributed storage cluster. The method includes: in response to a service request, obtaining the health status of each of at least two distributed storage clusters; simultaneously importing data into the at least two distributed storage clusters; and determining, by comparing the health statuses of the at least two distributed storage clusters, the distributed storage cluster to be accessed by the thread of the service request, so as to obtain the data corresponding to the service request from the distributed storage cluster to be accessed.

Description

Technical Field

[0001] This application relates to, but is not limited to, information technology in financial technology (Fintech), and particularly to a method, apparatus, device, and storage medium for accessing a distributed storage cluster. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, Fintech also places higher demands on technology. In the Fintech field, due to the huge user base and frequent transactions of internet banks, massive amounts of transaction data are generated. Therefore, the core transaction system of banks faces the challenge of storing, processing, and querying this massive amount of transaction data. In related technologies, the storage strategy for such massive amounts of transaction data is: recent (generally 10 days) transaction data is stored in the core transaction database, while historical transaction data is stored in a distributed storage cluster.

[0003] To ensure the consistency and accuracy of historical transaction data, the relevant technologies adopt a scheme of building two distributed storage clusters, one primary and one backup. This results in problems such as low utilization of the backup distributed storage cluster and long failover time. Summary of the Invention

[0004] In view of this, the embodiments of this application provide a method, apparatus, device, and storage medium for accessing a distributed storage cluster to solve at least one problem existing in the related technology, which can solve the problems of low utilization and long failover time of HBase cluster.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] On one hand, embodiments of this application provide a method for accessing a distributed storage cluster, the method comprising:

[0007] In response to a business request, the health status of each of at least two distributed storage clusters is obtained; the at least two distributed storage clusters simultaneously import data.

[0008] By comparing the health of the at least two distributed storage clusters, the distributed storage cluster to be accessed by the thread of the business request is determined from the at least two distributed storage clusters, so as to obtain the data corresponding to the business request from the distributed storage cluster to be accessed.

[0009] In another aspect, embodiments of this application provide a method for accessing a distributed storage cluster, the method comprising:

[0010] Receive probe requests from clients;

[0011] In response to the probe request, the health information table maintained by the controller is sent to the client so that the client can determine the distributed storage cluster to be accessed;

[0012] The health information table shall record at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0013] In another aspect, embodiments of this application provide an apparatus for accessing a distributed storage cluster, the apparatus comprising:

[0014] The acquisition module is used to acquire the health status of each of at least two distributed storage clusters in response to a business request; the at least two distributed storage clusters import data simultaneously.

[0015] The determination module is used to determine, by comparing the health of the at least two distributed storage clusters, the distributed storage cluster to be accessed by the thread of the business request, so as to obtain the data corresponding to the business request from the distributed storage cluster to be accessed.

[0016] In another aspect, embodiments of this application provide an apparatus for accessing a distributed storage cluster, the apparatus comprising:

[0017] The receiving module is used to receive probe requests from clients;

[0018] The sending module is used to respond to the probe request by sending the health information table maintained by the controller to the client, so that the client can determine the distributed storage cluster to be accessed; the health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0019] In another aspect, embodiments of this application provide a device for accessing a distributed storage cluster, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above-described method.

[0020] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0021] In this embodiment, the health status of each of at least two distributed storage clusters is obtained; data is simultaneously imported into the at least two distributed storage clusters; by comparing the health status of the at least two distributed storage clusters, the distributed storage cluster to be accessed by the thread of the business request is determined from the at least two distributed storage clusters, so as to obtain the data corresponding to the business request from the distributed storage cluster to be accessed. Thus, by simultaneously importing data into the at least two distributed storage clusters during implementation, data consistency is guaranteed. In this embodiment, there is no primary and backup distinction between the at least two distributed storage clusters; that is, this embodiment abandons the scheme of having two distributed storage clusters as primary and backup in related technologies. By obtaining and comparing the health status of each distributed storage cluster, the distributed storage cluster to be accessed can be dynamically selected, solving the timeliness problem of frequent storage cluster switching caused by inappropriate selection of the distributed storage cluster to be accessed, reducing the number of times the distributed storage cluster to be accessed is switched, improving the utilization efficiency of the storage cluster, and thus fundamentally solving the problems of low utilization and long fault switching time of the backup distributed storage cluster in related technologies. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the implementation process of the method for accessing a distributed storage cluster according to an embodiment of this application;

[0023] Figure 2 This is a schematic diagram illustrating the implementation process of the method for accessing a distributed storage cluster according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram illustrating the implementation process of the method for accessing a distributed storage cluster according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the implementation process of the method for accessing a distributed storage cluster according to an embodiment of this application;

[0026] Figure 5A This is a schematic diagram of the network architecture of a distributed storage cluster in related technologies;

[0027] Figure 5B This is a schematic diagram of the network architecture of the distributed storage cluster in an embodiment of this application;

[0028] Figure 6A This is a schematic diagram of the structural composition of the apparatus for accessing a distributed storage cluster according to an embodiment of this application;

[0029] Figure 6B This is a schematic diagram of the structural composition of the apparatus for accessing a distributed storage cluster according to an embodiment of this application;

[0030] Figure 7This is a schematic diagram of a hardware entity of a device that accesses a distributed storage cluster in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0035] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] Before introducing the embodiments of this application, a network architecture for a distributed storage cluster will be first described, such as... Figure 5B As shown, the client (HBase-client) issuing the business request connects to two distributed storage clusters (HBase-1 and HBase-2) simultaneously. A timed thread compares the health of the Connections on the two clusters and selects the cluster with the best health as the storage cluster used by the HBase-client's business thread. This improves the availability and automatic failover of the HBase-client.

[0037] See also Figure 5BAs shown, the business data flow in the core transaction system, or core transaction database (core transaction DB), is as follows: 1) Write the core real-time transaction data to the Kafka message center; 2) Use two Spark Streaming instances to consume the transaction detail data from the Kafka message center and write it to different HBase clusters (HBase-1 and HBase-2) simultaneously; 3) Write the progress of consuming data to the health information table of each HBase cluster to facilitate the comparison of the data synchronization progress of the real-time storage cluster by the HBase-Client.

[0038] Here, the HBase-client, which issues the business request, responds to the request by obtaining the health status of each of at least two distributed storage clusters. Data is simultaneously imported into the at least two distributed storage clusters. By comparing the health statuses of the at least two distributed storage clusters, the distributed storage cluster to be accessed by the thread making the business request is determined, and the data corresponding to the business request is retrieved from the cluster to be accessed. It should be noted that... Figure 5B This is one application scenario in the embodiments of this application, used to help understand this application, and does not limit this application.

[0039] This application provides a method for accessing a distributed storage cluster. Figure 1 This is a schematic diagram illustrating the implementation flow of the method for accessing a distributed storage cluster according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0040] Step S101: In response to a business request, obtain the health status of each of the at least two distributed storage clusters; the at least two distributed storage clusters import data simultaneously.

[0041] The at least two distributed storage clusters are deployed in a multi-active manner across data centers;

[0042] Here, the business request can be a request to obtain data required for business operations in a fintech application scenario. For example, a request to obtain data from a distributed storage cluster.

[0043] Here, the distributed storage cluster can be selected according to the application scenario. For example, the distributed storage cluster can be an HBase cluster. HBase is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a personal computer. This embodiment will use an HBase cluster as an example for explanation.

[0044] Here, health refers to the degree to which the HBase cluster can respond normally to business requests. This health can be measured by health levels. For example, health levels level-0, level-1, level-2, level-3, and level-4 can be used to represent the health of the HBase cluster, where level-0 represents the initial health level of the HBase cluster, and the relationship between the HBase cluster health levels is: level-0 > level-1 > level-2 > level-3 > level-4. When a business request requires the use of an HBase cluster, the optimal HBase cluster can be selected by comparing the health levels of the HBase clusters.

[0045] Here, the at least two distributed storage clusters are deployed in a multi-active mode across data centers. That is, at least two HBase clusters can be placed in different regions or in different computer rooms in the same region. When multiple HBase clusters are used at the same time, the connected HBase clusters are determined by performing health checks on each HBase cluster. If a connected HBase cluster fails, other fault-free HBase clusters can be used to replace the faulty HBase cluster to respond to business requests.

[0046] Step S102: By comparing the health of the at least two distributed storage clusters, determine the distributed storage cluster that the thread of the service request needs to access from the at least two distributed storage clusters, so as to obtain the data corresponding to the service request from the distributed storage cluster to be accessed.

[0047] For example, such as Figure 5B As shown, the client (HBase-client) issuing the business request connects to both HBase-1 and HBase-2 HBase clusters simultaneously. There is no primary or backup distinction between HBase-1 and HBase-2. The health status includes at least two health levels, for example, five health levels: level-0, level-1, level-2, level-3, and level-4. The relationship between the HBase cluster health levels is: level-0 > level-1 > level-2 > level-3 > level-4. If the health status of HBase-1 cluster is determined to be health level-0 and the health status of HBase-2 cluster is determined to be health level-1, then HBase-1 cluster is selected as the storage cluster connected to by the client issuing the business request.

[0048] In this embodiment, the health status of each of at least two distributed storage clusters is obtained; data is simultaneously imported into the at least two distributed storage clusters; by comparing the health status of the at least two distributed storage clusters, the distributed storage cluster to be accessed by the thread of the business request is determined from the at least two distributed storage clusters, so as to obtain the data corresponding to the business request from the distributed storage cluster to be accessed. Thus, in this embodiment, there is no primary and backup distinction between the at least two distributed storage clusters. That is, this embodiment abandons the scheme of having two distributed storage clusters as primary and backup in related technologies. By obtaining and comparing the health status of each distributed storage cluster, the distributed storage cluster to be accessed can be dynamically selected, solving the timeliness problem of frequent storage cluster switching caused by inappropriate selection of the distributed storage cluster to be accessed, reducing the number of times the distributed storage cluster to be accessed is switched, improving the utilization efficiency of the storage cluster, and thus fundamentally solving the problems of low utilization and long fault switching time of the backup distributed storage cluster in related technologies.

[0049] This application provides a method for accessing a distributed storage cluster. Figure 2 This is a schematic diagram illustrating the implementation flow of the method for accessing a distributed storage cluster according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes:

[0050] Step S210: In response to the business request, obtain the health assessment data of each of the at least two distributed storage clusters;

[0051] Here, the health assessment data can be one of the following: indicators of cluster response to business requests, cluster operation status indicators, and cluster data operation status indicators.

[0052] The indicators for cluster response to business requests may include one of the following: the number of abnormal requests within a preset time window, the percentage of abnormal requests within a preset time window, the number of requests whose response time exceeds a first time threshold within a preset time window, and the percentage of requests whose response time exceeds the first time threshold within a preset time window; the indicators for cluster operation status may include one of the following: the number of shard servers in a downtime state, and the number of shard servers in a disconnected state; the indicators for cluster data operation status may include one of the following: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0053] Step S220: Based on the health assessment data of each distributed storage cluster, determine the health of each distributed storage cluster according to a preset health assessment strategy.

[0054] The at least two distributed storage clusters are deployed in a multi-active manner across data centers;

[0055] Here, the health assessment data is the data collected when the storage cluster is running normally.

[0056] Here, the preset health assessment strategy may include at least one of the following: an assessment strategy based on the cluster response to business request status, an assessment strategy based on cluster operation status indicators, and an assessment strategy based on data operation status indicators.

[0057] The evaluation strategy based on the status of cluster response business requests is set based on at least one of the following evaluation parameters: the number of abnormal requests within a preset time window, the proportion of abnormal requests within a preset time window, and the number of requests whose response time exceeds a first time threshold within a preset time window.

[0058] The evaluation strategy based on cluster operation status indicators is set based on at least one of the following evaluation parameters: the number of shard servers in a downtime state and the number of shard servers in a disconnected state.

[0059] The evaluation strategy based on data operation status indicators is set based on at least one of the following evaluation parameters: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is being maintained.

[0060] In some embodiments, after obtaining health assessment data, the client uses this data as a parameter in a preset health assessment strategy to assess the health of each distributed storage cluster. For example, the client (HBase-Client) collects relevant health assessment data and, based on the collected data, determines the cluster health according to a preset health assessment strategy.

[0061] Step S230: By comparing the health of the at least two distributed storage clusters, determine the distributed storage cluster that the thread of the service request needs to access from the at least two distributed storage clusters, so as to obtain the data corresponding to the service request from the distributed storage cluster to be accessed.

[0062] In some embodiments, step S210, obtaining health assessment data for each of the at least two distributed storage clusters, includes: step S211A, probing the controller of the distributed storage cluster through a timed thread, and reading at least one of the following health assessment data: the number of shard servers in a downtime state, and the number of shard servers in a disconnected state.

[0063] Here, the HBase-Client can use a timed thread to monitor the HBase cluster's Master node and read metrics from shard servers (dead servers) that are down. If the number of dead servers is greater than 0, the shard server is in an abnormal down state. Alternatively, the HBase-Client can use a timed thread to probe the HBase cluster's Master node and read metrics from the shard servers (Region Servers). If a Region Server goes offline, it is considered to be in an abnormal offline state.

[0064] In some embodiments, step S210, obtaining health assessment data for each of the at least two distributed storage clusters, includes: step S211B, probing the controller (Master node) of the distributed storage cluster through a timed thread, reading the health information table, wherein the health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0065] For example, the health information table is shown in Table 1. It can be configured with the boolean keyword `is_bulkloading` to indicate whether the distributed cluster is currently importing data in bulk; the data type keyword `data_offset` to indicate the progress of synchronization to the distributed storage cluster; the boolean keyword `is_major_compaction` to indicate whether the distributed cluster is currently merging data; and the boolean keyword `is_switch` to indicate whether the distributed storage cluster is currently under maintenance.

[0066] Table 1 Cluster Health Information Table

[0067]

[0068] In this embodiment, the health status of each distributed storage cluster can be determined based on the health assessment data and a predefined assessment strategy. This allows the client to collect real-time health assessment data and dynamically determine the cluster's real-time health status according to the preset health assessment strategy.

[0069] This application provides a method for accessing a distributed storage cluster. Figure 3 This is a schematic diagram illustrating the implementation flow of the method for accessing a distributed storage cluster according to an embodiment of this application, as shown below. Figure 3 As shown, the method includes:

[0070] Step S310: In response to a business request, determine at least one data dimension associated with the preset health assessment strategy;

[0071] Here, the data dimensions may include: indicators of cluster response to business requests, cluster operation status indicators, and cluster data operation status indicators.

[0072] Step S320: Based on the at least one data dimension, collect health assessment data for each of the distributed storage clusters;

[0073] For example, 1) When the data dimension is an indicator of the cluster's response to business requests, the health assessment data collected for each of the distributed storage clusters may include at least one of the following: the number of abnormal requests within a preset time window, the proportion of abnormal requests within a preset time window, and the number of requests whose response time exceeds a first time threshold within a preset time window; 2) When the data dimension is a cluster operation status indicator, the health assessment data collected for each of the distributed storage clusters may include at least one of the following: the proportion of requests whose response time exceeds a first time threshold within a preset time window, and the number of shard servers that are down; 3) When the data dimension is a cluster data operation status indicator, the health assessment data collected for each of the distributed storage clusters may include at least one of the following: the number of shard servers that are offline, whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0074] Step S330: Based on the health assessment data of each distributed storage cluster, determine the health of each distributed storage cluster according to a preset health assessment strategy.

[0075] The at least two distributed storage clusters are deployed in a multi-active manner across data centers;

[0076] Step S340: By comparing the health of the at least two distributed storage clusters, determine the distributed storage cluster that the thread of the service request needs to access from the at least two distributed storage clusters, so as to obtain the data corresponding to the service request from the distributed storage cluster to be accessed.

[0077] In this embodiment of the application, health assessment data of each of the distributed storage clusters is collected based on the at least one data dimension. In this way, multi-dimensional health assessment data of each distributed storage cluster can be obtained, and the health of each distributed storage cluster can be judged according to the multi-dimensional preset health assessment strategy. The storage cluster with the highest health is selected as the storage cluster used by the thread of the business request, thereby reducing the number of times the business switches storage clusters and improving business processing efficiency.

[0078] This application provides a method for accessing a distributed storage cluster, the method comprising:

[0079] Step S410: In response to a business request, obtain health assessment data for each of at least two distributed storage clusters; the health assessment includes at least two health levels.

[0080] Step S420: Obtain the initial health level set for each of the distributed storage clusters;

[0081] The at least two distributed storage clusters are deployed in a multi-active manner across data centers;

[0082] For example, level-0 is used to represent the initial health level of the HBase cluster, and the relationship of the HBase cluster health levels is: level-0>level-1>level-2>level-3>level-4.

[0083] Step S430: Based on the health assessment data of each distributed storage cluster, determine the number of health levels that need to be reduced for the corresponding distributed storage cluster according to the preset health assessment strategy.

[0084] For example, if the preset health assessment strategy is based on the cluster's response to business requests, the number of health levels that the distributed storage cluster needs to be lowered can be set to 1. If the preset health assessment strategy is based on cluster operational status indicators, the number of health levels that the distributed storage cluster needs to be lowered can be set to 2. If the preset health assessment strategy is based on data operation status indicators, the number of health levels that the distributed storage cluster needs to be lowered can be set to 3.

[0085] Step S440: Determine the target health level of the distributed storage cluster based on the initial health level of each distributed storage cluster and the corresponding number of health levels that need to be reduced.

[0086] During implementation, based on the initial health level of level-0, the target health level is calculated by reducing the health level by the required number of levels. For example, if the initial health level is level-0 and the health level is reduced by 3 levels, the target health level is level-3.

[0087] Step S450: By comparing the health of the at least two distributed storage clusters, determine the distributed storage cluster that the thread of the service request needs to access from the at least two distributed storage clusters, so as to obtain the data corresponding to the service request from the distributed storage cluster to be accessed.

[0088] In this embodiment of the application, the health of the cluster can be determined according to the set initial health level and the preset health evaluation strategy, and the appropriate storage cluster for the thread to use as the business request can be dynamically determined according to the health of multiple clusters.

[0089] This application provides a method for accessing a distributed storage cluster, the method comprising:

[0090] Step S510: In response to the business request, obtain the health assessment data of each of the at least two distributed storage clusters;

[0091] Step S520: Based on the health assessment data of each distributed storage cluster, determine the health of each distributed storage cluster according to a preset health assessment strategy.

[0092] Step S530: By comparing the health of the at least two distributed storage clusters, determine the distributed storage cluster that the thread of the service request needs to access from the at least two distributed storage clusters, so as to obtain the data corresponding to the service request from the distributed storage cluster to be accessed.

[0093] In some embodiments, the preset health assessment strategy is set based on at least one of the following assessment parameters:

[0094] The following metrics are used to measure the number of abnormal requests within a preset time window, the percentage of abnormal requests within a preset time window, the percentage of responses received within a preset time window that take longer than the first time threshold, the number of responses received within a preset time window that take longer than the first time threshold, the number of shard servers that are down, the number of shard servers that are offline, whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0095] In some embodiments, the health status includes at least two health levels. When the evaluation parameters are: the number of abnormal requests within a preset time window, the proportion of abnormal requests within a preset time window, the proportion of responses received within a preset time window with a time consumption greater than a first time consumption threshold, and the number of responses received within a preset time window with a time consumption greater than the first time consumption threshold, the preset health status evaluation strategy includes one of the following:

[0096] 11) Based on the number of abnormal requests within the preset time window, determine the number of health levels that the distributed storage cluster needs to be downgraded according to a preset first mapping relationship; the first mapping relationship represents the correspondence between the number of abnormal requests within the preset time window and each health level.

[0097] Here, the preset time window is a pre-defined time period for initiating statistical business requests. For example, the preset statistical time period is 10 seconds, and the preset time window is 10 seconds. Abnormal requests include requests to the HBase cluster that do not receive a response within the preset time period.

[0098] Generally, the health level of the cluster is determined based on the time elapsed within a client-set time window and a threshold for anomaly requests. In other words, if the number of anomaly requests exceeds the threshold within a preset time window, the health level of the distributed storage cluster needs to be lowered by one level, from level-0 to level-1. For example, according to a preset mapping relationship, if the number of anomaly requests T exceeds the threshold within a preset time window, the health level is lowered by one, meaning the distributed storage cluster's health level is determined to be lowered from level-0 to level-1.

[0099] It should be noted that during implementation, multiple health assessment strategies can be configured simultaneously. However, if one health assessment strategy is active, the health level can be set to level-1. Generally, based on the number of abnormal requests within a preset time window, the cluster health level is typically lowered by one level. In this embodiment, since the number of health levels is relatively small (e.g., 5, from level-0 to level-4), a decrease of one health level is used as an example. Those skilled in the art can set more health levels, such as 20. In other embodiments, multiple thresholds for abnormal requests can be set, and then multiple health levels can be lowered based on the number of abnormal requests within a preset time window.

[0100] 12) Based on the proportion of abnormal requests within a preset time window, and according to a preset second mapping relationship, determine that the distributed storage cluster needs to be downgraded by at least one health level; the second mapping relationship represents the correspondence between the proportion of abnormal requests within the preset time window and each health level.

[0101] For example, based on the percentage of abnormal requests within a preset time window in the time consumption statistics, if the percentage of abnormal requests exceeds 0.1 within 10 seconds, it is determined that the distributed storage cluster needs to be downgraded from level-0 to level-1.

[0102] 13) Based on the number of responses received within a preset time window that take longer than a first time threshold, the distributed storage cluster is determined to need to be downgraded by at least one health level according to a preset third mapping relationship; the third mapping relationship represents the correspondence between the number of responses received within a preset time window that take longer than the first time threshold and each health level.

[0103] For example, based on the time consumption statistics, if the number of requests with a response time greater than 1 second within 10 seconds exceeds C1, then the distributed storage cluster needs to be downgraded from level-0 to level-1. For example, based on the time consumption statistics, if the number of business requests with a response time greater than 2 seconds within 10 seconds exceeds C2, then the distributed storage cluster needs to be downgraded from level-0 to level-1.

[0104] 14) Based on the proportion of responses received within a preset time window that take longer than the first time threshold, determine the number of health levels that the distributed storage cluster needs to be downgraded according to a preset fourth mapping relationship. The fourth mapping relationship represents the correspondence between the proportion of responses received within a preset time window that take longer than the first time threshold and each health level.

[0105] In some embodiments, the health status includes at least two health levels, and the evaluation parameters are: the number of shard servers in a downtime state and the number of shard servers in a disconnected state. The preset health status evaluation strategy includes one of the following:

[0106] 21) Based on the number of shard servers that are down, and according to the mapping relationship between the number of shard servers and each health level, determine the number of health levels that the distributed storage cluster needs to be downgraded.

[0107] Here, HBase-Client can periodically monitor the Master node of the HBase cluster and read the dead-server metric. If dead-server > 0, the server is in an abnormal state of failure. For example, HBase-Client monitors the Master node of the HBase cluster and reads the dead-server metric. If dead-server > 0, and the number of shard servers in a failure state is ≥ T3, then the distributed storage cluster needs to be downgraded by two health levels; this cluster is in a level-3 unhealthy state.

[0108] 22) Based on the number of shard servers that are offline, determine the number of health levels that the distributed storage cluster needs to be downgraded according to the mapping relationship between the number of shard servers and each health level.

[0109] Here, HBase-Client uses a timed thread to probe the Master node of the HBase cluster and read shard server metrics. If a shard server goes offline, it is in an abnormal offline state. For example, if HBase-Client uses a timed thread to probe the Master node of the HBase cluster and reads shard server metrics, and if the number of offline shard servers is ≥T4, then the distributed storage cluster needs to be downgraded by two health levels, and this cluster is in a level-3 unhealthy state.

[0110] In some embodiments, the health status includes at least three health levels, and the evaluation parameters are: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance. The preset health status evaluation strategy includes one of the following:

[0111] 31) If data is being imported into the distributed storage cluster in batches, the health level of the distributed storage cluster will be reduced by at least two health levels; if data is not being imported into the distributed storage cluster in batches, the health level of the distributed storage cluster will be maintained.

[0112] Here, a boolean keyword `is_bulkloading` can be set to indicate whether the current distributed cluster is performing batch data import. If `is_bulkloading = False`, no batch data import operation is being performed; if `is_bulkloading = True`, batch data import is being performed. For example, if importing data poses a risk of data inconsistency for the storage cluster used by the thread of the business request, the health level of the distributed storage cluster will be lowered by at least two health levels; this cluster will be in a level-2 unhealthy state.

[0113] 32) If the distributed storage cluster is merging data, the health level of the distributed storage cluster will be reduced by at least two health levels; if the distributed storage cluster is not merging data, the health level of the distributed storage cluster will be maintained.

[0114] Here, a boolean keyword `is_major_compaction` can be set to indicate whether the distributed cluster is currently merging data. If `is_major_compaction = False`, no data is being merged; the default value for this keyword is False. If `is_major_compaction = True`, the cluster is undergoing a major merge. For example, if a major merge is in progress, the storage cluster used by the threads providing business requests needs to be switched to another cluster to provide services. Because the major merge itself consumes relevant cluster resources, the health level of the distributed storage cluster will be lowered by at least two health levels; this cluster will be in a level-2 unhealthy state.

[0115] 33) If the distributed storage cluster is under maintenance, the health level of the distributed storage cluster shall be reduced by at least three health levels; if the distributed storage cluster is not under maintenance, the health level of the distributed storage cluster shall be maintained.

[0116] Here, a boolean keyword `is_switch` can be set to indicate whether the distributed storage cluster is under maintenance. If `is_switch = False`, it is not under maintenance; if `is_switch = True`, it is under maintenance. For example, if it is under maintenance and cannot provide service, the health level of the distributed storage cluster will be lowered by at least three health levels; this cluster will be in a level-4 unhealthy state.

[0117] In some embodiments, determining the distributed storage cluster to be accessed by the thread of the service request from the at least two distributed storage clusters by comparing their health status, and then obtaining the data corresponding to the service request from the distributed storage cluster to be accessed, includes:

[0118] By comparing the health of the at least two distributed storage clusters, the distributed storage cluster with the best health among the at least two distributed storage clusters is determined as the distributed storage cluster to be accessed by the thread of the business request.

[0119] If there are two or more distributed storage clusters with the best health, the distributed storage cluster with the best health shall be selected as the candidate cluster.

[0120] Obtain the current synchronization progress of each candidate cluster, and determine the candidate cluster with the highest synchronization progress as the distributed storage cluster to be accessed by the thread of the business request.

[0121] Generally, the keyword `data_offset` for a data type can be set to a data offset to indicate the progress of synchronization with the distributed storage cluster. The purpose of `data_offset` is that when the health of two clusters is at the same level, the timeliness of data synchronization can be monitored, thereby providing more real-time data services.

[0122] In this embodiment, by using evaluation parameters of different dimensions and evaluation strategies under different evaluation parameters, the health level of the cluster can be determined in scenarios where different types of evaluation data are obtained. This makes the method for accessing distributed storage clusters applicable to different application scenarios, improving the generalization of the method.

[0123] During implementation, such as Figure 5B As shown, the client (HBase-client) issuing the business request connects to two distributed storage clusters (HBase-1 and HBase-2) simultaneously. A timed thread is started to compare the health of the Connections between the two clusters, selecting the cluster with the best health as the storage cluster used by the HBase-client business thread. This improves the availability and automatic failover of the HBase-client. During the health comparison, the client needs to obtain the health assessment data for each of the distributed storage clusters; based on the health assessment data of each distributed storage cluster, and according to a preset health assessment strategy, the health of each distributed storage cluster is determined.

[0124] In some embodiments, the client probes the controller of the distributed storage cluster through a timed thread and reads at least one of the following health assessment data: the number of shard servers that are down or the number of shard servers that are offline.

[0125] In other embodiments, the client probes the controller of the distributed storage cluster through a timed thread and reads a health information table. The health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0126] Here, the distributed storage cluster includes a controller, which is used to receive probe requests from HBase-client and respond to the probe requests by sending health information tables and other health assessment data maintained by the controller to the HBase-client.

[0127] This application provides a method for accessing a distributed storage cluster. Figure 4 This is a schematic diagram illustrating the implementation flow of the method for accessing a distributed storage cluster according to an embodiment of this application, as shown below. Figure 4 As shown, the method includes:

[0128] Step S401: The controller of the distributed storage cluster receives a probe request from the client.

[0129] Here, the probe request can be sent periodically. The probe request is used to access the controller of the storage cluster and read the health information table of the HBase cluster from the controller.

[0130] Step S402: In response to the probe request, the health information table maintained by the controller is sent to the client so that the client can determine the distributed storage cluster to be accessed; the health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0131] For example, the health information table is shown in Table 1. It can be configured with the Boolean keyword `is_bulkloading` to indicate whether the current distributed cluster is importing data into the distributed storage cluster in bulk; the data type keyword `data_offset` to indicate the progress of synchronization to the distributed storage cluster; the Boolean keyword `is_major_compaction` to indicate whether the distributed storage cluster is merging data; and the Boolean keyword `is_switch` to indicate whether the distributed storage cluster is under maintenance.

[0132] In some embodiments, the method further includes:

[0133] Step S403: Based on whether to import data into the distributed storage cluster in batches, write the corresponding variable into the first field of the health information table maintained by the controller;

[0134] Here, a boolean keyword `is_bulkloading` can be set to indicate whether the current distributed cluster is importing data into the distributed storage cluster in bulk. When `is_bulkloading = False`, no bulk data import operation is performed; when `is_bulkloading = True`, a bulk data import operation is performed.

[0135] Step S404: Receive the data synchronization progress report from the message consumption tool, and write the value corresponding to the data synchronization progress into the second field of the health information table maintained by the controller; the data type of the second field is numeric.

[0136] Here, the message consumption tool can be real-time streaming data processing (Sparkstreaming). Sparkstreaming can accept data sources from Kafka, and the processed data can be stored in a file system database.

[0137] Here, you can set the keyword `data_offset` for the data type. It is a data offset used to represent the progress of data synchronization. It is a numeric type, and the numeric value represents the progress of data synchronization.

[0138] Step S405: Based on whether the distributed storage cluster is merging data, write the corresponding variable into the third field of the health information table maintained by the controller;

[0139] Here, you can set the boolean keyword `is_major_compaction` to indicate whether the current distributed cluster is merging data. If `is_major_compaction = False`, no data is being merged; the default value of this keyword is False. If `is_major_compaction = True`, the cluster is performing a major merge.

[0140] Step S406: Based on whether the distributed storage cluster is under maintenance, write the corresponding variable into the fourth field of the health information table maintained by the controller;

[0141] Here, you can set the boolean keyword `is_switch` to indicate whether the distributed storage cluster is being maintained. If `is_switch = False`, it is not being maintained; if `is_switch = True`, it is being maintained.

[0142] The data types of the first field, the third field, and the fourth field are all Boolean.

[0143] In some embodiments, the method further includes:

[0144] Step S407: Before importing data into the distributed storage cluster in batches, write a true value to the first field of the health information table maintained by the controller; after the batch import of data into the distributed storage cluster is completed, write a false value to the first field of the health information table maintained by the controller.

[0145] Step S408: Before merging data in the distributed storage cluster, write a true value to the third field of the health information table maintained by the controller; after merging data in the distributed storage cluster is completed, write a false value to the third field of the health information table maintained by the controller.

[0146] Step S409: Before the maintenance of the distributed storage cluster, write a true value to the fourth field of the health information table maintained by the controller; after the maintenance of the distributed storage cluster is completed, write a false value to the fourth field of the health information table maintained by the controller.

[0147] Here, steps S407 to S409 are used to update the fields corresponding to the operation after the operation is performed, so as to ensure that the health information table can reflect the real-time status of the HBase cluster.

[0148] In some embodiments, each of the distributed storage clusters is synchronized by consuming message data from different subscription messaging systems using different message consumption tools, and the different subscription messaging systems obtain message data from the same data source; the at least two distributed storage clusters synchronize data from the same data warehouse tool.

[0149] For example, two Spark Streaming instances can be used to consume transaction detail data from the Kafka message center and write it to different HBase clusters simultaneously.

[0150] In some embodiments, the progress of consuming data needs to be written to the data_offset field of the health information table of each HBase cluster, so as to facilitate the comparison of the data synchronization progress of the real-time storage cluster by the HBase-Client.

[0151] Here, the data warehouse tool (Hive) can be used to extract, transform, and load data. The data transfer tool can be Sqoop, which can extract transaction data from the database for day T-1 into the Hive cluster.

[0152] In this embodiment, each distributed storage cluster is synchronized by consuming message data from different subscription messaging systems using different message consumption tools, and these different subscription messaging systems obtain message data from the same data source. The at least two distributed storage clusters synchronize data using the same data warehouse tool, and the data warehouse tool updates data from the data source using the same data transfer tool. This eliminates the need for database replication synchronization mechanisms in related technologies, and by simultaneously importing data from the subscription messaging systems into each distributed storage cluster, it avoids data inconsistency issues that may result from replication.

[0153] This application provides a method for accessing a distributed storage cluster, the method comprising:

[0154] Step S41: The controller of the distributed storage cluster receives the client's probe request;

[0155] Step S42: In response to the probe request, the health information table maintained by the controller is sent to the client so that the client can determine the distributed storage cluster to be accessed; the health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0156] Step S43: Before importing data into the distributed storage cluster in batches, write a true value to the first field of the health information table maintained by the controller; after the batch import of data into the distributed storage cluster is completed, write a false value to the first field of the health information table maintained by the controller.

[0157] Step S44: Before merging the data in the distributed storage cluster, write a true value to the third field of the health information table maintained by the controller; after the data merging in the distributed storage cluster is completed, write a false value to the third field of the health information table maintained by the controller.

[0158] Step S45: Before the maintenance of the distributed storage cluster, write a true value to the fourth field of the health information table maintained by the controller; after the maintenance of the distributed storage cluster is completed, write a false value to the fourth field of the health information table maintained by the controller.

[0159] In some embodiments, there is no sequential order between steps S43, S44 and S45, and there is no sequential order between performing step S42 and performing steps S43 to S45.

[0160] In some embodiments, the method further includes:

[0161] Step S46: Detect whether the total amount of data synchronized by the distributed storage cluster within a preset time period is consistent with the total amount of data in the corresponding time period in the data warehouse tool;

[0162] Step S47: In response to the consistency of the total amount of data, sample the data in the data warehouse tool within the preset time period, and compare the sampled data with the corresponding data in the distributed storage cluster;

[0163] Step S48: In response to the inconsistency of the total data volume, and / or the inconsistency of the comparison, update the data volume within the preset time period through the batch import tool.

[0164] Here, steps S46 to S48 compare the data in terms of both quantity and value, and update the data in a timely manner when they are different, thus ensuring the correctness and consistency of the data.

[0165] In this embodiment of the application, a data verification mechanism is added to ensure the correctness and consistency of the data, thereby avoiding data loss and errors.

[0166] Because internet banks have a huge user base and users make frequent transactions, they generate massive amounts of transaction data. Therefore, the core transaction system of banks faces the challenge of storing, processing, and querying this massive amount of transaction data.

[0167] The core trading system typically uses a relational database (MySQL / Oracle) with transaction support for storage. However, relational databases in this technology cannot meet the needs of storing massive amounts of transaction data. Furthermore, because these relational databases have high requirements for storage hardware (generally SSDs), the storage cost is also high.

[0168] In related technologies, the storage strategy for such massive amounts of transaction data is as follows: recent (generally 10 days) transaction data is stored in the core transaction database, while historical transaction data is stored in a distributed database (Hadoop Database, HBase). Because HBase is a key-value (KV) database built on the HDFS file system, it can linearly expand to handle the increasing volume of transaction details over time. Secondly, HBase can use ordinary hard disk drives as storage hardware, thus controlling storage costs.

[0169] Transaction data is a core asset of banks and needs to be distributed across multiple data centers while ensuring eventual consistency and accuracy. Therefore, for HBase clusters, a multi-active solution across multiple data centers is required to meet these requirements. Related technologies for multi-active solutions between HBase clusters include... Figure 5A As shown:

[0170] Firstly, the following issues exist during HBase replication synchronization:

[0171] 1) Data consistency is compromised in certain abnormal situations.

[0172] In HBase replication, data changes are pushed to the standby cluster by reading the Write-Ahead Log (WAL) from each RegionServer. HBase maintains a queue of WAL files in ZooKeeper, allowing these files to be read in creation time order. However, when a Region is moved or a RegionServer fails over, the WAL logs on the new RegionServer may be pushed to the standby cluster before the old ones. In this case, the data write order on the standby cluster is inconsistent with that on the primary cluster. In more extreme cases, if this inconsistency occurs with the same data, it could lead to permanent inconsistencies between the primary and standby clusters.

[0173] For example, first, a Put operation is executed in the primary cluster, and then a Delete operation is executed to delete data in the primary cluster. However, the Delete operation is first replicated to the backup cluster. If the backup cluster performs a major compaction operation before receiving the Put operation, the major compaction operation will delete the delete marker. Subsequently, the backup cluster receives the Put operation, so this Put operation will not have a chance to execute the Delete operation on the backup cluster. The data deleted in the primary cluster will continue to exist on the backup cluster.

[0174] 2) Replication cannot support the synchronization of bulk-loaded data in the cluster.

[0175] When importing large amounts of data into an HBase cluster, HBase provides Bulkload technology for handling large volumes of data. Bulkload directly generates HBase storage files (HFiles) from the data, bypassing the HBase RegionServer component. This avoids the generation of WAL files, and therefore data cannot be synchronized via replication.

[0176] 3) In the event of a cluster failure, after switching the failed cluster to the standby cluster, there may be inconsistencies in the data between the failed cluster and the standby cluster.

[0177] If a cluster fails and the replication synchronization process is incomplete, switching to a standby cluster will result in the absence of relevant data that was not fully synchronized.

[0178] Secondly, HBase clusters suffer from low utilization and long failover times, which manifests in the following two aspects:

[0179] 1) From the perspective of HBase-Client usage, if one HBase cluster is unavailable, it is necessary to switch to another HBase cluster, which can easily lead to a waste of resources in the other cluster.

[0180] 2) In the event of a cluster failure, Spark Streaming that synchronizes data needs to be switched to a backup cluster, which results in a long switching time when the cluster fails.

[0181] To address the aforementioned issues, this application provides a method for increasing HBase reliability, comprising: (1) removing the replication synchronization mechanism by importing data into two clusters simultaneously, thereby avoiding data inconsistency issues that may result from replication; (2) adding a data verification mechanism to ensure data correctness and consistency, thereby avoiding data loss and inconsistency; and (3) transforming the HBase-Client into a client that supports multi-cluster mode and implementing cluster health checks, dynamically selecting available clusters through a policy comparator, thereby achieving high-performance service and avoiding failover time issues.

[0182] HBase-Client is used to collect data to assess the health of the HBase cluster. Based on the collected data, a pre-defined strategy is applied to determine the health of the HBase cluster. When a business request requires the use of the HBase cluster, the optimal HBase cluster is selected by comparing its health status.

[0183] First, the health levels of an HBase cluster are represented by level-0, level-1, level-2, and level-3. Level-0 represents the initial health level of the HBase cluster, and the relationship between the HBase cluster health levels is: level-0 > level-1 > level-2 > level-3 > level-4. Determining the health level of an HBase cluster involves at least the following three methods:

[0184] (1) Determine whether the HBase cluster health is at level-1 based on the time window duration and exception threshold set by HBase-Client, including:

[0185] a) Determine the health level of the HBase cluster based on the time consumption statistics of the aforementioned time window:

[0186] For example, according to the time consumption statistics, if the percentage of abnormal requests with a response time of more than 1 second that are received within 10 seconds exceeds 0.1%, the HBase cluster connected to the HBase-Client is in a level-1 unhealthy state.

[0187] b) Determine the health level of the HBase cluster based on the anomaly statistics of the aforementioned time window:

[0188] For example, according to the anomaly statistics, if the percentage of abnormal requests exceeds 0.1 within 10 seconds, the HBase cluster connected to the HBase-Client is in a level-1 unhealthy state.

[0189] c) Determine the health level of the HBase cluster based on the time consumption statistics of each business connection to the HBase cluster from the HBase-Client:

[0190] Here, each business can be a part of a complete business. For example, according to the time consumption statistics, in every business request within 10 seconds, the proportion of HBase requests with a response time greater than 2 seconds exceeds 0.2%, indicating that the HBase cluster connected to the HBase-Client is in a level-1 unhealthy state.

[0191] In this embodiment of the application, the health of the HBase cluster is judged by configuring the strategy in multiple dimensions, such as time window, exception threshold, exception type, exception quantity, exception percentage and time consumption dimension; as well as the timeout percentage and time consumption statistics of each HBase request; and the timeout percentage and time consumption statistics of multiple HBase requests in the entire business.

[0192] (2) Detecting HBase cluster health indicators (health level) based on HBase-Client

[0193] a) HBase-Client uses a timed thread to monitor the Master node of the HBase cluster and read the dead-server metric. If dead-server > 0, the cluster is in a level-2 unhealthy state.

[0194] b) HBase-Client uses a timed thread to probe the Master node of the HBase cluster and read Region Server metrics. If a Region Server goes offline, the cluster is in a level-2 unhealthy state.

[0195] (3) Tables representing the health of different HBase clusters were pre-defined to achieve the optimal solution for cluster health.

[0196] a) Define a table representing cluster health in different HBase clusters.

[0197] Here, health_info is used to mark the table name, and Rowkey is used to mark the cluster name. The cluster name is used to locate the cluster.

[0198] Second, real-time data is simultaneously written to different HBase clusters.

[0199] 1) Write the core real-time transaction data to the Kafka message center.

[0200] 2) Use two Spark Streaming instances to consume transaction detail data from the Kafka message center and write it to different HBase clusters simultaneously.

[0201] 3) The progress of consuming data is written to the data_offset field of the health_info table of each HBase cluster, which facilitates the comparison of the data synchronization progress of the real-time storage cluster by the HBase-Client.

[0202] Third, regularly verify the consistency and accuracy of daily data.

[0203] 1) Use Sqoop to extract the transaction data of day T-1 from the database in the core transaction system into the Hive cluster;

[0204] 2) Set is_bulkloading to true in the health_info table of the HBase cluster that needs to be checked (see the subsequent description of HBase-Client modification), and then compare the total amount of data in the HBase table for the day with the data in Hive.

[0205] 3) If the total number can be aligned, compare the sampled detailed data in Hive with the HBase cluster to see if they are consistent. If the sampled details are not aligned, push the corresponding table back to the HBase cluster via bulkload.

[0206] 4) If the total number is not aligned, the corresponding table will be pushed back to the HBase cluster via bulkload.

[0207] 5) Once all operations are complete, set is_bulkloading in the health_info of this HBase cluster to false.

[0208] 6) After completing the data consistency and correctness verification and completion actions, in order to provide better query services for the HBase cluster, it is generally necessary to enable major_compaction.

[0209] 7) Set the is_major_compaction table of the HBase cluster's health_info table to true to enable the large-scale merging of this HBase cluster.

[0210] 8) Enable scheduled checks to see if the major merging service is complete. If it is complete, set is_major_compaction in the health_info table to false.

[0211] 9) After completing steps 1 to 8 above, repeat steps 1 to 8 to complete the data consistency and correctness checks and merging operations of another HBase cluster.

[0212] 10) Start a scheduled thread to scan the cluster health information table (health_info table) of each HBase cluster. The scan condition is the name of the cluster.

[0213] 11) When is_switch=true, this cluster is in a level-3 unhealthy state.

[0214] When is_bulkloading=true or is_major_compaction=true, this cluster is in a level-2 unhealthy state.

[0215] Here, the behaviors of multiple subsystems (such as Spark Streaming and Hive) that affect the HBase cluster are abstracted into a table. The relevant behaviors are written to the HBase cluster table by different subsystems, and the HBase-client uses the data in the table to determine the health of the cluster, thus providing the best HBase cluster service quality.

[0216] Fourth, when an HBase-client's business request requires the use of HBase, the health of HBase is assessed by executing a preset health assessment strategy, and the optimal HBase cluster is selected by comparing the health scores.

[0217] In some embodiments, such as Figure 5B As shown, the HBase-client connects to two HBase clusters simultaneously. A scheduled thread compares the health levels of the Connections on the two clusters, selecting the cluster with the highest health level (level-0, level-1, level-2, level-3) as the storage cluster used by the HBase-client's business threads. This improves the availability and automatic failover of the HBase-client.

[0218] Based on the foregoing embodiments, this application provides an apparatus for accessing a distributed storage cluster. The apparatus includes various modules and sub-modules included in each module, which can be implemented by a processor in a client device of the storage cluster; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0219] Figure 6A This is a schematic diagram of the structural composition of the apparatus for accessing a distributed storage cluster according to an embodiment of this application, as shown below. Figure 6AAs shown, the device 600A includes an acquisition module 601 and a determination module 602, wherein:

[0220] The acquisition module 601 is used to acquire the health status of each of at least two distributed storage clusters in response to a business request; the at least two distributed storage clusters import data simultaneously.

[0221] The determination module 602 determines the distributed storage cluster to be accessed by the thread of the business request from the at least two distributed storage clusters by comparing their health status, so as to obtain the data corresponding to the business request from the distributed storage cluster to be accessed.

[0222] In some embodiments, the acquisition module 601 includes: an acquisition submodule and a determination submodule, wherein: the acquisition submodule is used to acquire health assessment data of each of the at least two distributed storage clusters; and the determination submodule is used to determine the health of each distributed storage cluster based on the health assessment data of each distributed storage cluster and according to a preset health assessment strategy.

[0223] In some embodiments, the acquisition submodule includes: a first determining unit and a collecting unit, wherein: the first determining unit is used to determine at least one data dimension associated with the preset health assessment strategy; and the collecting unit is used to collect health assessment data for each of the distributed storage clusters based on the at least one data dimension.

[0224] In some embodiments, the acquisition submodule is configured to probe the controller of the distributed storage cluster through a timed thread and read at least one of the following health assessment data: the number of shard servers in a downtime state and the number of shard servers in a disconnected state.

[0225] In some embodiments, the acquisition submodule is used to probe the controller of the distributed storage cluster via a timed thread and read the health information table. The health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0226] In some embodiments, the determining submodule includes: an acquisition unit, a second determining unit, and a third determining unit, wherein: the acquisition unit is used to acquire the initial health level set for each of the distributed storage clusters; the second determining unit is used to determine the number of health levels that need to be reduced for the corresponding distributed storage cluster based on the health assessment data of each of the distributed storage clusters and according to a preset health assessment strategy; and the third determining unit is used to determine the target health level of the distributed storage cluster based on the initial health level of each of the distributed storage clusters and the corresponding number of health levels that need to be reduced.

[0227] In some embodiments, the preset health assessment strategy is set based on at least one of the following assessment parameters:

[0228] The following metrics are used to measure the number of abnormal requests within a preset time window, the percentage of abnormal requests within a preset time window, the number of requests whose response time exceeds the first time threshold within a preset time window, the percentage of requests whose response time exceeds the first time threshold within a preset time window, the number of shard servers that are down, the number of shard servers that are offline, whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0229] In some embodiments, the health status includes at least two health levels, and the preset health status assessment strategy includes one of the following:

[0230] Based on the number of abnormal requests within the preset time window, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset first mapping relationship; the first mapping relationship represents the correspondence between the number of abnormal requests within the preset time window and each health level.

[0231] Based on the proportion of abnormal requests within a preset time window, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset second mapping relationship; the second mapping relationship represents the correspondence between the proportion of abnormal requests within the preset time window and each health level.

[0232] Based on the number of responses received within a preset time window that take longer than a first time threshold, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset third mapping relationship; the third mapping relationship represents the correspondence between the number of responses received within a preset time window that take longer than the first time threshold and each health level.

[0233] Based on the proportion of responses received within a preset time window that take longer than a first time threshold, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset fourth mapping relationship. The fourth mapping relationship represents the correspondence between the proportion of responses received within a preset time window that take longer than the first time threshold and each health level.

[0234] In some embodiments, the preset health assessment strategy includes one of the following:

[0235] Based on the number of shard servers that are down, and according to the mapping relationship between the number of shard servers and each health level, the number of health levels that the distributed storage cluster needs to be downgraded is determined.

[0236] Based on the number of shard servers that are offline, and according to the mapping relationship between the number of shard servers and each health level, the number of health levels that the distributed storage cluster needs to be downgraded is determined.

[0237] In some embodiments, the health status includes at least three health levels, and the preset health status assessment strategy includes:

[0238] If data is being imported into the distributed storage cluster in batches, the health level of the distributed storage cluster will be lowered by at least two health levels; if no data is being imported into the distributed storage cluster in batches, the health level of the distributed storage cluster will be maintained.

[0239] If the distributed storage cluster is merging data, the health level of the distributed storage cluster will be reduced by at least two health levels; if the distributed storage cluster is not merging data, the health level of the distributed storage cluster will be maintained.

[0240] If the distributed storage cluster is under maintenance, its health level will be lowered by at least three health levels; if the distributed storage cluster is not under maintenance, its health level will be maintained.

[0241] In some embodiments, the determining module is configured to determine the distributed storage cluster with the best health among the at least two distributed storage clusters as the distributed storage cluster to be accessed by the thread of the business request by comparing the health of the at least two distributed storage clusters.

[0242] If there are two or more distributed storage clusters with the best health, the distributed storage cluster with the best health shall be selected as the candidate cluster.

[0243] Obtain the current synchronization progress of each candidate cluster, and determine the candidate cluster with the highest synchronization progress as the distributed storage cluster to be accessed by the thread of the business request.

[0244] Based on the foregoing embodiments, this application provides an apparatus for accessing a distributed storage cluster. The apparatus includes various modules and sub-modules included in each module, which can be implemented by a processor in the storage cluster; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0245] Figure 6B This is a schematic diagram of the structural composition of the apparatus for accessing a distributed storage cluster according to an embodiment of this application, as shown below. Figure 6B As shown, the device 600B includes a receiving module 610 and a transmitting module 620, wherein:

[0246] The receiving module 610 is used to receive probe requests from the client;

[0247] The sending module 620 is used to respond to the probe request by sending the health information table maintained by the controller to the client, so that the client can determine the distributed storage cluster to be accessed; the health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

[0248] In some embodiments, the device 600B further includes a first update module, a second update module, a third update module, and a fourth update module, wherein: the first update module is configured to write a corresponding variable into a first field of a health information table maintained by the controller based on whether data is being imported in batches into the distributed storage cluster; the second update module is configured to receive a data synchronization progress report from a message consumption tool and write a value corresponding to the data synchronization progress into a second field of the health information table maintained by the controller; the data type of the second field is a numeric type; the third update module is configured to write a corresponding variable into a third field of the health information table maintained by the controller based on whether the distributed storage cluster is merging data; the fourth update module is configured to write a corresponding variable into a fourth field of the health information table maintained by the controller based on whether the distributed storage cluster is being maintained; wherein the data types of the first field, the third field, and the fourth field are Boolean types.

[0249] In some embodiments, the device 600B further includes a fifth update module, a sixth update module, and a seventh update module, wherein: the fifth update module is configured to write a true value to a first field of the health information table maintained by the controller before batch importing data into the distributed storage cluster; and to write a false value to the first field of the health information table maintained by the controller after the batch import of data into the distributed storage cluster is completed; the sixth update module is configured to write a true value to a third field of the health information table maintained by the controller before merging data in the distributed storage cluster; and to write a false value to the third field of the health information table maintained by the controller after the data merging in the distributed storage cluster is completed; the seventh update module is configured to write a true value to a fourth field of the health information table maintained by the controller before maintaining the distributed storage cluster; and to write a false value to the fourth field of the health information table maintained by the controller after the maintenance of the distributed storage cluster is completed.

[0250] In some embodiments, the device 600B includes a detection module, a sampling module, and an eighth update module, wherein: the detection module is used to detect whether the total amount of data synchronized by the distributed storage cluster within a preset time period is consistent with the total amount of data in the corresponding time period in the data warehouse tool; the sampling module is used to sample the data in the data warehouse tool within the preset time period in response to the data total amount being consistent, and compare the sampled data with the corresponding data in the distributed storage cluster; the eighth update module is used to update the amount of data within the preset time period through a batch import tool in response to the data total amount being inconsistent, and / or, in the case of inconsistent comparison.

[0251] In some embodiments, each of the distributed storage clusters is synchronized by different message consumption tools consuming message data from different subscription message systems, and the different subscription message systems obtain message data from the same data source;

[0252] The at least two distributed storage clusters synchronize data from the same data warehouse tool.

[0253] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0254] It should be noted that, in the embodiments of this application, if the above-described method for accessing a distributed storage cluster is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device (which may be a personal computer, server, etc.) accessing a distributed storage cluster to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0255] Correspondingly, embodiments of this application provide a device for accessing a distributed storage cluster (including a client device and a distributed storage cluster), comprising a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above-described method.

[0256] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0257] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0258] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a device accessing a distributed storage cluster in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein...

[0259] The processor 701 typically controls the overall operation of the device 700.

[0260] Communication interface 702 enables the device to communicate with other terminals or servers over a network.

[0261] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed by the processor 701 and the various modules in the device 700 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0262] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0263] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0264] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0265] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0266] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0267] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0268] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device accessing a distributed storage cluster (which may be a personal computer, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0269] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for accessing a distributed storage cluster, characterized in that, The method, applied to a client of a storage cluster, includes: In response to a business request, health assessment data for each of at least two distributed storage clusters is obtained; based on the health assessment data for each distributed storage cluster, the health level of each distributed storage cluster is determined according to a preset health assessment strategy; data is imported simultaneously from the at least two distributed storage clusters; the health level includes at least two health grades, and the preset health assessment strategy includes: based on the number of abnormal requests within a preset time window, determining the number of health grades that the distributed storage cluster needs to be lowered according to a preset first mapping relationship; the first mapping relationship represents the correspondence between the number of abnormal requests within the preset time window and each health grade; By comparing the health of the at least two distributed storage clusters, the distributed storage cluster with the best health is determined as the distributed storage cluster to be accessed by the thread of the business request; if there are more than two distributed storage clusters with the best health, the distributed storage cluster with the best health is selected as a candidate cluster; the current synchronization progress of each candidate cluster is obtained, and the candidate cluster with the highest synchronization progress is determined as the distributed storage cluster to be accessed by the thread of the business request.

2. The method according to claim 1, characterized in that, The step of obtaining health assessment data for each of at least two distributed storage clusters includes: Determine at least one data dimension associated with the preset health assessment strategy; Based on the at least one data dimension, health assessment data for each of the distributed storage clusters is collected.

3. The method according to claim 1, characterized in that, The step of obtaining health assessment data for each of the at least two distributed storage clusters includes: probing the controller of the distributed storage cluster through a timed thread and reading at least one of the following health assessment data: the number of shard servers in a downtime state and the number of shard servers in a disconnected state. or, The step of obtaining health assessment data for each of the at least two distributed storage clusters includes: probing the controller of the distributed storage cluster through a timed thread, reading a health information table, wherein the health information table records at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

4. The method according to any one of claims 1 to 3, characterized in that, The health status includes at least two health levels. The determination of the health status of each distributed storage cluster based on its health status assessment data and according to a preset health assessment strategy includes: Obtain the initial health level set for each of the distributed storage clusters; Based on the health assessment data of each distributed storage cluster, and in accordance with the preset health assessment strategy, the number of health levels that need to be reduced for the corresponding distributed storage cluster is determined. The target health level of the distributed storage cluster is determined based on the initial health level of each distributed storage cluster and the corresponding number of health levels that need to be reduced.

5. The method according to any one of claims 1 to 3, characterized in that, The preset health assessment strategy includes one of the following: Based on the proportion of abnormal requests within a preset time window, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset second mapping relationship. The second mapping relationship represents the correspondence between the proportion of abnormal requests within the preset time window and each health level; Based on the number of responses received within a preset time window that take longer than a first time threshold, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset third mapping relationship. The third mapping relationship represents the correspondence between the number of responses received within a preset time window that take longer than the first time threshold and each health level. Based on the proportion of responses received within a preset time window that take longer than a first time threshold, the number of health levels that the distributed storage cluster needs to be downgraded is determined according to a preset fourth mapping relationship. The fourth mapping relationship represents the correspondence between the proportion of responses received within a preset time window that take longer than the first time threshold and each health level.

6. The method according to any one of claims 1 to 3, characterized in that, The preset health assessment strategy includes one of the following: Based on the number of shard servers that are down, and according to the mapping relationship between the number of shard servers and each health level, the number of health levels that the distributed storage cluster needs to be downgraded to is determined. Based on the number of shard servers that are offline, and according to the mapping relationship between the number of shard servers and each health level, the number of health levels that the distributed storage cluster needs to be downgraded is determined.

7. The method according to any one of claims 1 to 3, characterized in that, The health status includes at least three health levels, and the preset health assessment strategy includes: If data is being imported into the distributed storage cluster in batches, the health level of the distributed storage cluster will be lowered by at least two health levels; if no data is being imported into the distributed storage cluster in batches, the health level of the distributed storage cluster will be maintained. If the distributed storage cluster is merging data, the health level of the distributed storage cluster will be reduced by at least two health levels; if the distributed storage cluster is not merging data, the health level of the distributed storage cluster will be maintained. If the distributed storage cluster is under maintenance, its health level will be lowered by at least three health levels; if the distributed storage cluster is not under maintenance, its health level will be maintained.

8. A method for accessing a distributed storage cluster, characterized in that, The method includes a controller applied to each of at least two distributed storage clusters, the method comprising: Receive probe requests from clients; In response to the probe request, the health information table maintained by the controller is sent to the client, so that the client can determine the health of each distributed storage cluster according to the health assessment data recorded in the health information table and a preset health assessment strategy. By comparing the health of the at least two distributed storage clusters, the distributed storage cluster with the best health is determined as the distributed storage cluster to be accessed by the thread of the business request. If there are more than two distributed storage clusters with the best health, the distributed storage cluster with the best health is selected as a candidate cluster. The current synchronization progress of each candidate cluster is obtained, and the candidate cluster with the highest synchronization progress is determined as the distributed storage cluster to be accessed by the thread of the business request. The health status includes at least two health levels, and the preset health status assessment strategy includes: based on the number of abnormal requests within a preset time window, determining the number of health levels that the distributed storage cluster needs to be downgraded according to a preset first mapping relationship; the first mapping relationship represents the correspondence between the number of abnormal requests within the preset time window and each health level. The health information table shall record at least one of the following health assessment data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

9. The method according to claim 8, characterized in that, The method further includes: Based on whether to import data into the distributed storage cluster in batches, write the corresponding variable into the first field of the health information table maintained by the controller; Upon receiving the data synchronization progress report from the message consumption tool, a value corresponding to the data synchronization progress is written into the second field of the health information table maintained by the controller; the data type of the second field is a numeric type. Based on whether the distributed storage cluster is merging data, the corresponding variable is written into the third field of the health information table maintained by the controller; Based on whether the distributed storage cluster is under maintenance, the corresponding variable is written into the fourth field of the health information table maintained by the controller; The data types of the first field, the third field, and the fourth field are all Boolean.

10. The method according to claim 9, characterized in that, The method further includes: Before importing data into the distributed storage cluster in batches, a true value is written to the first field of the health information table maintained by the controller; after the batch import of data into the distributed storage cluster is completed, a false value is written to the first field of the health information table maintained by the controller. Before the distributed storage cluster merges the data, a true value is written to the third field of the health information table maintained by the controller; after the distributed storage cluster completes the data merging, a false value is written to the third field of the health information table maintained by the controller. Before the maintenance of the distributed storage cluster, a true value is written to the fourth field of the health information table maintained by the controller; after the maintenance of the distributed storage cluster is completed, a false value is written to the fourth field of the health information table maintained by the controller.

11. The method according to claim 8, characterized in that, Each of the distributed storage clusters is synchronized by different message consumption tools consuming message data from different subscription message systems, and the different subscription message systems obtain message data from the same data source; The at least two distributed storage clusters synchronize data from the same data warehouse tool.

12. The method according to claim 10 or 11, characterized in that, The method further includes: The system detects whether the total amount of data synchronized by the distributed storage cluster within a preset time period is consistent with the total amount of data in the corresponding time period in the data warehouse tool. In response to the condition that the total amount of data is consistent, the data within the preset time period in the data warehouse tool is sampled, and the sampled data is compared with the corresponding data in the distributed storage cluster; In response to inconsistencies in the total amount of data, and / or inconsistencies in comparison, the amount of data within the preset time period is updated using a batch import tool.

13. An apparatus for accessing a distributed storage cluster, characterized in that, The device includes: The acquisition module is used to acquire the health status of each of at least two distributed storage clusters in response to a business request; the at least two distributed storage clusters import data simultaneously. The determination module is configured to compare the health of the at least two distributed storage clusters, determine the distributed storage cluster with the best health among the at least two distributed storage clusters as the distributed storage cluster to be accessed by the thread of the business request; if there are more than two distributed storage clusters with the best health, the distributed storage cluster with the best health is selected as a candidate cluster; obtain the current synchronization progress of each candidate cluster, and determine the candidate cluster with the highest synchronization progress as the distributed storage cluster to be accessed by the thread of the business request.

14. An apparatus for accessing a distributed storage cluster, characterized in that, The device includes: The receiving module is used to receive probe requests from clients; The sending module is configured to respond to the probe request by sending a health information table maintained by the controller to the client, so that the client can obtain the health status of each of the at least two distributed storage clusters according to the health information table; by comparing the health status of the at least two distributed storage clusters, the distributed storage cluster with the best health status is determined as the distributed storage cluster to be accessed by the thread of the business request; if there are more than two distributed storage clusters with the best health status, the distributed storage cluster with the best health status is selected as a candidate cluster; the current synchronization progress of each candidate cluster is obtained, and the candidate cluster with the highest synchronization progress is determined as the distributed storage cluster to be accessed by the thread of the business request; the health information table records at least one of the following health evaluation data: whether data is being imported into the distributed storage cluster in batches, the progress of synchronization to the distributed storage cluster, whether the distributed storage cluster is merging data, and whether the distributed storage cluster is under maintenance.

15. A device for accessing a distributed storage cluster, comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7, or the steps of the method according to any one of claims 8 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7, or implements the steps of the method according to any one of claims 8 to 12.

Images