A data disaster recovery method and device

Abstract

The application is suitable for the technical field of data storage, and provides a data disaster recovery method and device, comprising the following steps: determining a fault type in a distributed cluster system and a range of data to be recovered; acquiring state parameters of multiple normal nodes in the distributed cluster system, and service sensitivity features and data block access features of the data to be recovered; performing data preprocessing operations; respectively adopting a service sensitivity evaluation model and a node state evaluation model to determine a data recovery priority list and a node priority list; determining a fault processing strategy according to the data recovery priority list and the node priority list; and executing the fault processing strategy. Through the above method, the real-time evaluation algorithm of the heterogeneous big data node is increased on the basis of dynamically updating the data recovery priority based on the service sensitivity evaluation model, and compared with the traditional technology, the fault perception of the business system can be reduced while the data disaster recovery efficiency is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data storage, and particularly relates to a data disaster recovery method and device. BACKGROUND

[0002] In recent years, with the vigorous development of the big data era, the big data distributed cluster plays an increasingly important role as the storage basis of cloud computing. The file system of the distributed cluster often uses the Hadoop Distributed File System (HDFS) to perform mass data processing and stream batch processing. At the same time, due to the unreliability and bandwidth limitation of network connection, node failure is prone to occur in the distributed cluster. Under this background, it is necessary to develop data disaster recovery technology. At present, the data disaster recovery technology can check each node through failure detection, and realize the backup and recovery of data in the big data distributed cluster system through the means of recovering the data in the failed node. However, the existing technology has limited factors considered in the research scheme of the data disaster recovery technology, and has certain limitations, and the data disaster recovery capability of the cluster system is low. SUMMARY

[0003] Therefore, the embodiments of the application provide a data disaster recovery method and device, which can solve the problems of high business system failure perception and low data disaster recovery efficiency.

[0004] In a first aspect, the embodiments of the application provide a data disaster recovery method, which is applied to a distributed cluster system, the distributed cluster system includes a plurality of nodes, and the method includes the following steps.

[0005] When it is determined that a node in the distributed cluster system fails, the type of the failure and the range of data to be recovered are determined.

[0006] When the type of the failure is a data node failure type, a first evaluation data set and a second evaluation data set are obtained, wherein the first evaluation data set includes a business sensitivity feature and / or a data block access feature of the data to be recovered, and the second evaluation data set includes a state parameter of a node other than a failed data node in the plurality of nodes.

[0007] The first evaluation data set is preprocessed to obtain a preprocessed first evaluation data set, and the second evaluation data set is preprocessed to obtain a preprocessed second evaluation data set.

[0008] determine a data recovery priority list according to the pretreated first evaluation data set and a business sensitivity evaluation model, wherein the business sensitivity evaluation model is configured to output a priority of the data to be recovered according to the pretreated first evaluation data set, and the data recovery priority list is configured to represent a priority order of the data to be recovered;

[0009] determine a node priority list according to the pretreated second evaluation data set and a node state evaluation model, wherein the node state evaluation model is configured to evaluate real-time states of the nodes according to the pretreated second evaluation data set and output the node priority list, the node priority list includes at least one node, and the at least one node is a node whose state meets a preset condition in the plurality of nodes, and the node priority list is configured to represent a priority order of the at least one node;

[0010] determine a fault handling strategy according to the data recovery priority list and the node priority list, wherein the fault handling strategy is that the data to be recovered is sequentially sent to the at least one node according to the data recovery priority list;

[0011] execute the fault handling strategy.

[0012] In the embodiment of the application, after determining the fault type of the node as a data node fault and the range of the data to be recovered, the business sensitivity feature and / or the data block access feature of the data to be recovered and the data state parameters of the plurality of nodes except the fault data node are obtained, after preprocessing the obtained data, the business sensitivity evaluation model is called to dynamically update the data block recovery priority according to the pretreated data, and the node state evaluation model is called to evaluate the real-time state of the node in real time, so that the data block that needs to be recovered urgently can be sent to the node with the optimal state to perform the recovery operation. Compared with the prior art, the embodiment of the application considers the sensitivity of the business and the difference of the cluster nodes, can fully exert the advantages of the cluster, improves the recovery efficiency of the data block, and reduces the fault perception of the business system.

[0013] In a possible implementation, the business sensitivity feature of the data to be recovered includes one or more of the following features: a priority feature of a business system to which the data block belongs, a priority feature of a functional module to which the data block belongs, a business attribute feature of the data block, and a timeliness feature of the data block.

[0014] In a possible implementation, the business sensitivity evaluation model is configured to output a priority of the data to be recovered according to the pretreated first evaluation data set, including:

[0015] The business sensitivity evaluation model is used to output the priority of the to-be-restored data according to a hotness value feature of the to-be-restored data and a business sensitivity feature of the to-be-restored data.

[0016] The hotness value feature of the to-be-restored data is obtained by analyzing a data block access feature based on a neural network algorithm.

[0017] In a possible implementation, the data block access feature is determined according to one or more of the following parameters: a file name, a file operation type, a file operation time, and a file operation permission.

[0018] That is, by introducing the business sensitivity evaluation model to analyze and mine data, the data with high business sensitivity and in urgent need of recovery in a business system can be preferentially restored, so that the effect of reducing user fault perception can be achieved.

[0019] In a possible implementation, the node state evaluation model is used to evaluate real-time states of the nodes according to the preprocessed second evaluation data set, and output the node priority list, including:

[0020] The node state evaluation model is used to determine the node priority list according to state parameters of the nodes and historical state evaluation values of the nodes.

[0021] The state parameters include one or more of the following parameters: CPU utilization, memory usage, disk I / O usage, network bandwidth usage, command response time, command queue length, and disk usage.

[0022] Based on the node state evaluation model, the real-time state evaluation values of the nodes are updated in real time, which is beneficial to determine at least one node that can be used to perform a data recovery operation, and arrange priorities of the at least one node, so as to ensure that a node with the optimal state in the at least one node is used to process data with high business sensitivity and in urgent need of recovery. In this way, the cluster advantage can be greatly exerted, and the data disaster recovery efficiency can be improved.

[0023] In a possible implementation, the executing the fault processing strategy includes:

[0024] The to-be-restored data is sent to the at least one node in sequence for processing according to the priority order of the to-be-restored data and the priority order of the at least one node.

[0025] In a possible implementation, after the to-be-restored data is sent to the at least one node in sequence for processing according to the priority order of the to-be-restored data and the priority order of the at least one node, the method further includes:

[0026] If there is remaining data in the data to be recovered, the remaining data is sent to a node in the at least one node that has completed data processing, wherein the remaining data refers to data in the data to be recovered that has not been processed by the at least one node.

[0027] In a possible implementation, the preprocessing operation comprises one or more of the following operations:

[0028] Category feature processing, null value processing, and data normalization processing, wherein the category feature processing is used to convert non-numeric data into numeric data.

[0029] In a second aspect, an embodiment of the present application provides a data disaster recovery apparatus, configured to execute the method in the first aspect or any possible implementation of the first aspect. Specifically, the apparatus comprises modules (or units) configured to execute the method in the first aspect or any possible implementation of the first aspect.

[0030] In a third aspect, an embodiment of the present application provides a distributed cluster system comprising a plurality of nodes. The plurality of nodes mainly comprises two types of nodes, namely a name node and a data node. Optionally, the plurality of nodes can further comprise an auxiliary name node. The distributed cluster system is configured to execute the method for data disaster recovery in any implementation of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the method in the first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a data disaster recovery apparatus, causes the data disaster recovery apparatus to execute the method in the first aspect or any possible implementation of the first aspect.

[0033] Compared with the prior art, the embodiment of the present application has the beneficial effects that: the embodiment of the present application designs a data disaster recovery method for a distributed cluster system, maximally utilizes node resources through real-time evaluation algorithms of heterogeneous big data nodes, and effectively improves data disaster recovery efficiency. Meanwhile, the embodiment of the present application dynamically updates a data recovery priority based on a business sensitivity evaluation model, can reduce the fault perception of a business system, and can improve the robustness of a big data distributed cluster system and a business system and improve the service quality of a platform while recovering data through real-time evaluation of the nodes and the data recovery priority. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0035] Figure 1 is a schematic diagram of the application scenario of the data disaster recovery method provided by the embodiments of the present application;

[0036] Figure 2 is a flowchart of the data disaster recovery method provided by the embodiments of the present application;

[0037] Figure 3 is a schematic block diagram of the evaluation model provided by the embodiments of the present application;

[0038] Figure 4 is a flowchart of the method for determining the data recovery priority list provided by the embodiments of the present application;

[0039] Figure 5 is a flowchart of the method for determining the node priority list provided by the embodiments of the present application;

[0040] Figure 6 is a flowchart of the data preprocessing process provided by the embodiments of the present application;

[0041] Figure 7 is an example flowchart of the data disaster recovery method provided by the embodiments of the present application;

[0042] Figure 8 is a structural block diagram of the data disaster recovery device provided by the embodiments of the present application;

[0043] Figure 9 is a structural diagram of the data disaster recovery provided by the embodiments of the present application. DETAILED DESCRIPTION

[0044] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary details.

[0045] It should be understood that the word “comprise” or variations such as “comprises” or “comprising”, when used in this specification and in the accompanying claims, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0046] It should also be understood that the term “and / or” when used in this specification and in the following claims is to be interpreted as to include one or more of the listed items, as well as any combination of the listed items, and that the use of “and / or” is not limited to a particular combination of the listed items.

[0047] As used in this specification and in the claims, the terms “if’ and “when” can be interpreted to mean “upon” or “in response to a determination” or “in response to a detection” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be interpreted to mean “upon a determination” or “in response to a determination” or “upon detecting [a described condition or event]” or “in response to detecting [a described condition or event]”, depending on the context.

[0048] In addition, the terms “first”, “second”, “third”, etc. are used in the description and in the following claims merely to distinguish one element from another, and are not necessarily intended to imply relative importance.

[0049] Reference in the specification to “one embodiment” or “some embodiments” etc., means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase “in one embodiment” or “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms “including”, “comprising”, “having” and variations thereof are meant to encompass the terms “including but not limited to”, unless otherwise expressly specified.

[0050] Embodiments of the present application are applied to a distributed cluster system. The distributed cluster system includes a plurality of nodes, wherein the plurality of nodes in the distributed cluster system mainly include two types of nodes, namely a name node and a data node. Generally, the plurality of nodes can include one name node and a plurality of data nodes.

[0051] Optionally, the plurality of nodes can further include an auxiliary name node. The auxiliary node can back up the data of the name node to avoid data loss caused by the collapse of the name node.

[0052] In a possible implementation, the distributed cluster system can be an HDFS system. The HDFS system architecture adopts a middle control node architecture, and when data needs to be read or written, a name node storing metadata needs to be accessed first to obtain storage information of actual data, and then a data node storing the actual data is accessed for reading or writing.

[0053] For ease of understanding, before introducing the embodiments of the present application, some terms or concepts related to the embodiments of the present application are introduced briefly.

[0054] Node: can realize the functions of storing data, indexing and searching. Each node has a unique name as its identity. Multiple nodes can form a cluster.

[0055] Name node (or management node), mainly used for saving metadata information of a distributed cluster system (such as an HDFS system), maintaining the namespace of the file system, and responding to the read / write requirements of the client, and sending instructions to the data node. The name node includes a mirror file (Fslmage) for saving the directory tree of the file system, and an operation log (EditLog) for modifying the directory tree.

[0056] Data node, belonging to a software node running on a single machine in a distributed cluster system (such as an HDFS system), usually organized in the form of a rack. The data node is mainly used for storing actual data blocks and executing read / write operations on the data blocks.

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0058] In a possible application scenario, the data disaster recovery method provided by the embodiments of the present application can be applied to an application scenario involving a heterogeneous big data distributed cluster system.

[0059] The following will be described by way of example in combination with a specific scenario in Figure 1 .

[0060] Figure 1 is a schematic diagram of an application scenario of the data disaster recovery method provided by the embodiments of the present application, as shown in Figure 1 , the application scenario of the heterogeneous big data distributed cluster system includes a client, a name node and N data nodes. Optionally, the heterogeneous big data distributed cluster system can further include an auxiliary name node.

[0061] The client is used to split files to be written to HDFS, interact with the NameNode and DataNodes, and manage and access HDFS. For example, the client types include HDFS command line (HDFS Shell Command), Java API, C API Libhdfs, and other methods of accessing HDFS. The Java API uses Java code to operate and access HDFS. The C API Libhdfs is a C API for HDFS, pre-compiled in Hadoop distributions, and supports C language clients that can be used to manipulate HDFS and the file system.

[0062] The data nodes respond to read and write requests from clients, as well as to create, delete, and copy commands from name nodes. The data nodes create, delete, and copy data blocks stored on them. These data blocks are stored as files on the data nodes, and each data block includes the data itself and metadata, including the data length, block checksum, and timestamp. Depending on the data's access frequency, it can be stored on different storage media within the data nodes. For example, frequently accessed data can be stored on higher-performance storage media (such as memory or SSDs) to improve read and write performance; rarely accessed data can be stored on archive storage media to reduce storage costs.

[0063] To Figure 1 The functions or roles of the terms used in this document are briefly explained. It should be noted that... Figure 1 The names of the various terms shown are merely illustrative and the embodiments of this application are not limited thereto. In fact, the names of the various terms can be replaced with other names, as long as they have the corresponding functions.

[0064] Furthermore, for ease of description, the heterogeneous big data distributed cluster system will be referred to as a distributed cluster system in the following text.

[0065] In this application embodiment, when a node failure occurs in the distributed cluster system, data backup and recovery are required to improve the robustness of the cluster and business system. Therefore, this application proposes a data disaster recovery method that can improve data disaster recovery efficiency while reducing the fault perception of the business system.

[0066] The following will combine Figures 2 to 7 This application describes the process of performing data disaster recovery in its embodiments.

[0067] Figure 2is a flowchart of a data disaster recovery method provided by an embodiment of the present application. It should be understood that the flowchart is provided by way of example and not limitation, Figure 2 The method in the method can be applied to Figure 1 application scenarios shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps: Figure 2

[0068] Step S110: determining a fault type and a range of data to be recovered when determining that a node in the distributed cluster system fails.

[0069] Optionally, the fault type comprises a name node fault and a data node fault. For the data node fault, the present application does not limit the judgment manner of the data node fault.

[0070] Optionally, as a possible implementation manner, whether the data node fails can be judged by monitoring whether the data node sends heartbeat information to the name node within a preset time length.

[0071] In other words, from the perspective of the name node, if the heartbeat information is not received by the name node beyond a preset time threshold, it is determined that the fault type is the data node fault.

[0072] Exemplarily, the data node fault is caused by one or more of the following conditions:

[0073] (1) a data block stored in the data node lacks a copy;

[0074] (2) a data block is repeatedly duplicated due to a data block verification error;

[0075] (3) the name node cannot receive the heartbeat information of the data node due to network failure or data node failure.

[0076] To ensure the safety of the data stored on the data node, the heartbeat mechanism is used to monitor the data node. The working principle of the heartbeat mechanism is that after the HDFS cluster is started, the data node sends heartbeat packet registration information to the name node and periodically sends heartbeat information to the name node for failure monitoring. Generally, when the heartbeat information sent by the data node is not received by the name node for more than ten minutes, it is determined that the data node is invalid or fails.

[0077] Optionally, as a possible implementation manner, if the name node is in the safe mode for a long time, it is determined that the fault type is the name node fault.

[0078] In other words, if the name node is in the safe mode for a long time and cannot provide read-write function, i.e., the name node is in a state of being unable to respond to the read-write request of the client for a long time, it is determined that the fault type is the name node fault. ​

[0079] Exemplarily, the name node failure refers to a cold backup failure of the name node.

[0080] It can be understood that, when the auxiliary name node is deployed in the distributed cluster system, the name node failure refers to a failure of the name node itself or a failure of the auxiliary name node.

[0081] It can be understood that, before step S110, it can be monitored whether a node in the distributed cluster system fails. The specific manner of monitoring the node failure is not limited in the embodiments of the present application. For example, a failure detection module or a node state detection module can be set in the distributed cluster system to monitor whether a node fails.

[0082] Optionally, the range of the data to be recovered refers to a range of data to be recovered affected by the failed node. The range of the data to be recovered can also be referred to as a range of data blocks to be recovered.

[0083] Exemplarily, the range of the data to be recovered includes data damaged due to the failed node and data directly or indirectly affected due to the failed node.

[0084] Optionally, as a possible implementation manner, the range of the data to be recovered is determined by a default API component of the Hadoop component.

[0085] In another expression, after determining that the failure type is the data node failure, the list of data blocks stored on the failed data node is determined by the default API component of the Hadoop component.

[0086] Exemplarily, in some embodiments of the present application, the node failure of the distributed node is monitored and warned by an open-source big data cluster monitoring platform. Specifically, the open-source big data monitoring platform can monitor the node failure mainly characterized by the data node failure and the name node failure.

[0087] The open-source big data monitoring platform (WGCLOUD) is a software that can be automatically monitored and run after being decompressed. The WGCLOUD supports running on a Linux system, a Windows system, a Unix system, a MacOS system and other ARM, Android systems, etc.

[0088] The open-source big data monitoring platform monitors the name node and the data node in the distributed cluster system. When the open-source big data monitoring platform detects that a node fails, the type of the node failure is determined according to the monitored failure type.

[0089] When the node abnormality is monitored, the fault type of the node and the range of the data to be recovered are determined. If the fault type is a name node fault, the name node is directly caused to exit the safe mode. If the fault type is a data node fault, the range of the data to be recovered is obtained and step S120 is executed, and the cluster data disaster recovery process is immediately started. It should be understood that the above-mentioned manner of determining the name node fault is only an exemplary description, and the determination manner can also have other manners, for example, whether the starting time length of the name node exceeds a time threshold. The embodiments of the present application are not limited thereto.

[0090] Exemplarily, in some embodiments of the present application, the big data management platform can also be used to monitor and early warn the node fault of the distributed node. Specifically, the big data management platform can monitor the node fault mainly characterized by the data node fault and the name node fault.

[0091] The big data management platform is a Web-based tool supporting the supply, management and monitoring of Apache Hadoop clusters. At present, it has supported most Hadoop components, including a distributed storage system (HDFS), a distributed computing system (MapReduce), a data warehouse tool (Hive), a large-scale data analysis platform (Apache Pig), a distributed open source database (Hadoop Database, Hbase), a distributed application coordination service software (Zookeeper), a data migration tool (SQL to Hadoop, Sqoop) and a Hadoop table storage management tool (Hcatalog) and the like.

[0092] Step S120: When the fault type is a data node fault type, a first evaluation data set and a second evaluation data set are obtained, wherein the first evaluation data set includes the business sensitivity feature and / or the data block access feature of the data to be recovered, and the second evaluation data set includes the state parameters of the nodes except the fault data node in the plurality of nodes.

[0093] It should be understood that the state parameters of the nodes except the fault data node in the plurality of nodes refer to the state parameters of the normal data nodes except the fault data node in the distributed cluster system.

[0094] In some embodiments of the present application, when the fault type is determined to be a data node fault type, the first evaluation data set and the second evaluation data set in the distributed cluster system can be obtained through the open source big data monitoring platform. For example, assuming that the open source big data monitoring platform is used to monitor the distributed cluster system, the open source big data monitoring platform can monitor various indicators of the host in real time. For example, one or more indicators of each node monitored by the open source big data monitoring platform can be used to obtain the first evaluation data set and the second evaluation data set.

[0095] For example, the first evaluation data set includes one or more indicators such as process applications, files, ports, logs, data blocks, data tables, and other resources on the server.

[0096] For example, the second evaluation data set includes one or more indicators such as CPU usage, CPU temperature, memory usage, disk capacity, disk IO, hard disk SMART health status, system load, connection number, network card traffic, and hardware system.

[0097] It should be understood that the one or more indicators included in the first evaluation data set and the second evaluation data set described above are only examples of indicators monitored by the open source big data monitoring platform, and the embodiments of the present application are not limited thereto.

[0098] It should be understood that the first evaluation data set described above is only an example, and the first evaluation data set can also have other naming methods. The embodiments of the present application are not limited thereto.

[0099] It should be understood that the second evaluation data set described above is only an example, and the second evaluation data set can also have other naming methods. The embodiments of the present application are not limited thereto.

[0100] Step S130: pre-processing the first evaluation data set to obtain a pre-processed first evaluation data set, and pre-processing the second evaluation data set to obtain a pre-processed second evaluation data set.

[0101] During data collection, incomplete data collection, noise, inconsistent data types, and other problems often occur due to different considerations, human factors, or software problems. The collected data often contains null values, duplicates, independent or incorrect data points, and features that cannot be compared. Therefore, before data analysis, the data of multiple nodes obtained needs to be pre-processed to obtain pre-processed data for subsequent analysis.

[0102] Step S140: determining a data recovery priority list according to the preprocessed first evaluation data set and a business sensitivity evaluation model, wherein the business sensitivity evaluation model is used to output a priority of to-be-recovered data according to the preprocessed first evaluation data set, and the data recovery priority list is used to represent a priority order of the to-be-recovered data.

[0103] In the embodiment of the present application, the data block access feature and the business sensitivity feature in the preprocessed first evaluation data set are input into the business sensitivity evaluation model, the business sensitivity evaluation model performs prediction based on a business sensitivity comprehensive evaluation algorithm, and the data recovery priority list is output.

[0104] Optionally, the data block access feature includes one or more of the following features: a file name, a file operation type, a file operation time, and a file operation permission. After the data block access feature is determined, the data block access feature is collected and stored in real time.

[0105] Optionally, the business sensitivity feature of the to-be-recovered data includes one or more of the following features: a priority feature of a business system to which the data block belongs, a priority feature of a function module to which the data block belongs, a business attribute feature of the data block, and a timeliness feature of the data block.

[0106] Illustratively, the priority feature of the business system to which the data block belongs includes a business system with a high access frequency, a business system with a relatively high access frequency, a business system with a low access frequency, and a business system that is almost not accessed, and the like. The priority of the business system with the high access frequency is higher than the priority of the business system that is almost not accessed. The higher the priority of the business system to which the data block belongs corresponds, the more beneficial it is to assign a higher priority value to the data when the data recovery priority is comprehensively evaluated.

[0107] Illustratively, the priority feature of the function module to which the data block belongs includes a function module with a high use frequency, a function module with a relatively high use frequency, a function module with a low use frequency, and a function module that is almost not used, and the like. Similarly, the priority of the function module with the high use frequency is higher than the priority of the function module that is almost not used. The higher the priority of the function module to which the data block belongs corresponds, the more beneficial it is to assign a higher priority value to the data when the data recovery priority is comprehensively evaluated.

[0108] Illustratively, the business attribute feature of the data block includes process data and result data.

[0109] Illustratively, the timeliness feature of the data block includes data generation timeliness, which is usually years, quarters, months, days, hours, and the like.

[0110] The business sensitivity evaluation model receives the preprocessed first evaluation data set, first obtains data block access features and business sensitivity features of the data from the preprocessed first evaluation data set, then analyzes and mines the data block access features and the business sensitivity features of the data, and finally outputs a data recovery priority list, where the data recovery priority list is used to represent a data recovery priority list of a priority order of the to-be-recovered data.

[0111] Figure 3 The upper part in FIG. 1 shows a schematic block diagram of a business sensitivity evaluation model. As shown in the business sensitivity evaluation model, Figure 3 According to the priority features of the business system to which the data block belongs, the priority features of the function module to which the data block belongs, the business attribute features of the data block, the timeliness features of the data block, and the data hotness value, the business sensitivity is comprehensively evaluated to obtain a data recovery priority list. The data hotness value is obtained by calculating data block access features by using a neural network algorithm.

[0112] It should be understood that the above description of the business sensitivity evaluation model is only an example, and the business sensitivity evaluation model can also have other naming methods, for example, a business sensitivity comprehensive evaluation model. The embodiments of the present application are not limited thereto.

[0113] It should be understood that the above description of the data recovery priority list is only an example, and the data recovery priority list can also have other naming methods, for example, a data disaster recovery priority list. The embodiments of the present application are not limited thereto.

[0114] In the embodiments of the present application, after the business sensitivity evaluation model calculates the hotness value of the to-be-recovered data according to the preprocessed data, the data recovery priority list is determined on the premise of the business sensitivity. That is, by the way of preferentially recovering data with high business sensitivity and urgent recovery in the business system, the effect of reducing user fault perception can be achieved.

[0115] Step S150: determining a node priority list according to the preprocessed second evaluation data set and a node state evaluation model, where the node state evaluation model is used to evaluate real-time states of each node according to the preprocessed second evaluation data set and output the node priority list, the node priority list includes at least one node, the at least one node is a node whose state meets a preset condition in the plurality of nodes, and the node priority list is used to represent a priority order of the at least one node.

[0116] Optionally, as an implementation manner, the node satisfying the preset condition refers to a real-time state evaluation value of each node calculated according to the node state evaluation model, and on the basis of combining a historical state evaluation value of the node, a node with a state evaluation value greater than a preset threshold in the node is taken as a node for recovering data or processing data. The preset condition is that the state evaluation value is greater than the preset threshold.

[0117] It can be understood that the above examples of the preset condition are only exemplary descriptions, and the embodiments of the present application are not limited thereto. In fact, the preset condition can also be defined in other ways according to different needs. The embodiments of the present application do not make specific limitations here on the way of defining the preset condition.

[0118] The node state evaluation model is used to evaluate the real-time state of each node according to the preprocessed second evaluation data set and output the node priority list, wherein the node priority list contains at least one node satisfying a preset condition.

[0119] Figure 3 The lower part of FIG. 1 shows a schematic block diagram of a node state evaluation model. As shown in the lower part of FIG. 1, the node state evaluation model includes a node state parameter acquisition module 101, a node state evaluation module 102, a node state evaluation value storage module 103, a node state evaluation value updating module 104, a node state evaluation value historical storage module 105, a node state evaluation value real-time storage module 106, a node state evaluation value comprehensive evaluation module 107, and a node priority list output module 108. Figure 3 The node state evaluation model shown in the lower part of FIG. 1: according to the node state parameter, the real-time state evaluation value of the node is determined based on the state evaluation algorithm, and the business sensitivity comprehensive evaluation is performed on the real-time state evaluation value and the historical state evaluation value of the node, so that the data recovery priority list can be obtained. It should be understood that the above description of the node state evaluation model is only exemplary, and the node state evaluation model can also have other naming ways, for example, a big data distributed node comprehensive evaluation model. The embodiments of the present application are not limited thereto.

[0120] It should be understood that the above description of the node priority list is only exemplary, and the node priority list can also have other naming ways, for example, a state optimal node list. The embodiments of the present application are not limited thereto.

[0121] Based on the node state evaluation model, the real-time state evaluation value of the node is updated in real time, which is beneficial to determine the node that can be used to perform the data recovery operation, and arrange the priority of the node, so as to ensure that the node with the optimal state in the node receives the data with high business sensitivity and urgent recovery in the business system. Through the above-mentioned manner, the cluster advantage can be greatly exerted, and the data disaster recovery efficiency is improved.

[0122] Step S160: determining a fault handling strategy according to the data recovery priority list and the node priority list, the fault handling strategy referring to sequentially sending the data to be recovered to the at least one node according to the data recovery priority list.

[0123] It should be understood that the above description of the failure processing strategy is only exemplary, and the failure processing strategy can also be named in other ways, for example, a data disaster recovery backup strategy. The embodiments of the present application are not limited thereto.

[0124] Step S170: executing the failure processing strategy.

[0125] Optionally, as an implementation manner, the to-be-recovered data is sequentially sent to the at least one node for processing according to the priority order of the to-be-recovered data and the priority order of the at least one node.

[0126] Exemplarily, if the to-be-recovered data has three copies, the priority order of the to-be-recovered data is 0, 1, and 2 respectively, the at least one node includes node A, node B, and node C, and the priority order of the nodes is 0, 1, and 2 respectively, the data with the priority of 2 is first sent to node C for processing according to the priority order of the to-be-recovered data, the data with the priority of 1 is sent to node B for processing, and the data with the priority of 0 is last sent to node A for processing.

[0127] Optionally, as an implementation manner, after the to-be-recovered data is sequentially sent to the at least one node for processing according to the priority order of the to-be-recovered data, if there is still remaining data in the to-be-recovered data, the remaining data is sent to the node in the at least one node that has completed data processing, wherein the remaining data refers to the data in the to-be-recovered data that has not been processed by the at least one node.

[0128] In other words, the to-be-recovered data cannot be one-to-one corresponding to the at least one node in the node priority list. Therefore, part of the data that needs to be recovered urgently is first sent to the node to perform the recovery operation, and the remaining data is sequentially sent to the node that has completed data processing according to the priority order. In this way, the data disaster recovery efficiency can be improved.

[0129] Optionally, as an implementation manner, the flow of the recovery operation includes:

[0130] Step 1: checking and listing all data block states.

[0131] The specific manner of checking and listing all data block states is not limited in the embodiments of the present application.

[0132] In a possible implementation manner, the checking and listing of all data block states can be implemented in the following manner:

[0133] hdfs fsck file absolute path -files

[0134] It should be understood that the above is described only as an example, and the embodiments of the present application are not limited thereto.

[0135] The second step is to print the location information and the rack information of the data block.

[0136] It should be understood that the specific manner of checking and listing the status of all data blocks is not specifically limited in the embodiments of the present application.

[0137] In a possible implementation manner, the checking and listing of the status of all data blocks can be implemented in the following manner:

[0138] hdfs fsck file absolute path -files -blocks -locations

[0139] hdfs fsck file absolute path -files -blocks -locations -racks

[0140] It should be understood that the above is described only as an example, and the embodiments of the present application are not limited thereto.

[0141] The third step is to exit the safe mode of the name node.

[0142] It should be understood that the specific manner of checking and listing the status of all data blocks is not specifically limited in the embodiments of the present application.

[0143] In a possible implementation manner, the exiting of the safe mode of the name node can be implemented in the following manner:

[0144] hdfs dfsadmin -safemode leave NameNode

[0145] It should be understood that the above is described only as an example, and the embodiments of the present application are not limited thereto.

[0146] The fourth step is to set the storage location of the data block copy.

[0147] It should be understood that the specific manner of checking and listing the status of all data blocks is not specifically limited in the embodiments of the present application.

[0148] In a possible implementation manner, the checking and listing of the status of all data blocks can be implemented in the following manner:

[0149] chooseTargetInOrder sets the node of the copy storage

[0150] It should be understood that the above four steps are described only as an example of the flow of the recovery operation, and the embodiments of the present application are not limited thereto.

[0151] In the embodiment of the present application, by monitoring the nodes in the distributed cluster system, when it is determined that a node fails, the type of failure and the range of data to be recovered are determined; when the type of failure is a data node failure, the state parameters on multiple nodes and the business sensitivity features and data block access features of the data to be recovered are obtained; then, according to a business sensitivity evaluation model, a data recovery priority list is determined; and according to a node state evaluation model, a node priority list is determined, and according to the data recovery priority list and the node priority list, a failure handling strategy is determined and executed. Compared with the prior art, the present application dynamically updates the data recovery priority based on the business sensitivity evaluation model, and at the same time, by real-time evaluation of the nodes and the data recovery priority, the data recovery operation is performed, which can achieve the effect of sending the data blocks that need to be recovered urgently to the nodes with the best state to quickly perform the recovery operation, thereby achieving the effect of improving the recovery efficiency of the data blocks while reducing the failure perception of the business system.

[0152] For the convenience of those skilled in the art, the determination method of the data recovery priority list is described below in combination with specific examples in the Figure 4 It should be understood that the method in the Figure 4 is a refinement of step S140 in the data disaster recovery method shown in Figure 2 , and is not intended to limit the embodiments of the present application to the specific scenarios illustrated.

[0153] Optionally, as one possible embodiment, the determination of the data recovery priority list includes the following steps:

[0154] Step S141: According to the neural network algorithm and the data block access features in the preprocessed first evaluation data set, the hotness value of the data to be recovered is determined.

[0155] As an example but not as a limitation, in the embodiments of the present application, LSTM neural network algorithm can be used to calculate the hotness value of the data to be recovered. Wherein, according to the input data block access features, the LSTM neural network algorithm calculates the hotness value corresponding to the data to be recovered. The LSTM neural network algorithm has the advantages of being able to process long sequence data and obtain long distance data information, and can pass the information processed at the current moment to the next moment. Therefore, by using the LSTM neural network algorithm, the hotness value of the data in the future period of time can be predicted based on the current calculated data hotness value and the historical data hotness value, which can solve the problem that the HDFS heterogeneous storage strategy cannot be formulated because the access hotness of the file cannot be known in advance.

[0156] In a possible implementation, the data block access feature is determined according to one or more of the following parameters: a file name, a file operation type, a file operation time, and a file operation permission.

[0157] In a possible implementation, the hotness value of the data to be recovered can also be obtained through a recurrent neural network algorithm. The embodiments of the present application do not make specific limitations on the manner of determining the hotness value of the data to be recovered.

[0158] Step S142: determining the recovery priority of each data to be recovered according to the business sensitivity feature in the preprocessed first evaluation data set and the hotness value of the data to be recovered. For example, the embodiments of the present application perform business sensitivity comprehensive evaluation on the business sensitivity feature of the data to be recovered and the hotness value of the data to be recovered based on a multiple linear regression model, to determine the recovery priority of each data to be recovered.

[0159] Alternatively, in another possible implementation, a fuzzy comprehensive evaluation method, a multi-factor comprehensive evaluation method, or the like can be used to perform business sensitivity comprehensive evaluation on the business sensitivity feature of the data to be recovered and the hotness value of the data to be recovered.

[0160] It should be understood that the above is described by taking the multiple linear regression model as an example, and other models or algorithms can also be used to perform business sensitivity comprehensive evaluation on the business sensitivity feature of the data to be recovered and the hotness value of the data to be recovered, and the embodiments of the present application are not limited thereto.

[0161] Step S143: obtaining the data recovery priority list according to the recovery priority of the data to be recovered.

[0162] For example, the data to be recovered in the node is arranged according to the recovery urgency priority, the priority of the data urgently to be recovered is marked with a larger value, and the priority of the data to be recovered not urgently or slightly is marked with a smaller value.

[0163] In summary, after the business sensitivity evaluation model receives the preprocessed data, the data block access feature and the business sensitivity feature of the data to be recovered are obtained through the preprocessed first evaluation data set, the data block access feature and the business sensitivity feature of the data are analyzed and mined, and finally the data recovery priority list is output, where the data recovery priority list is used to represent the priority order of the data to be recovered. After obtaining the data recovery priority list, the data to be recovered can be sequentially sent to the node according to the priority order of the data to be recovered, and specifically the data urgently to be recovered can be sent to the node in the best state to perform the recovery operation, so that the beneficial effect of improving the data disaster recovery efficiency can be obtained.

[0164] For the convenience of those skilled in the art to understand, the determination method of the node priority list is described below in combination with specific examples in the following Figure 5 It should be understood that the method in Figure 5 is an example rather than a limitation, Figure 2 The refinement of S150 in the data disaster recovery method shown in

[0165] In one possible implementation, the determining the node priority list includes the following steps:

[0166] Step S151: calculating the real-time state evaluation value of the node according to the preprocessed second evaluation data set.

[0167] For example, in one possible implementation, the preprocessed second evaluation data set includes state parameters affecting the real-time performance of the node. According to the state parameters, the real-time state evaluation value of the node is calculated.

[0168] Specifically, the state parameters include one or more of the following parameters: CPU utilization, memory usage, disk I / O usage, network bandwidth usage, command response time, command queue length, and disk usage. After determining the state parameters affecting the real-time performance of the node, the state parameters are stored in real time.

[0169] For example but not limitation, the state evaluation value of each node can be obtained based on the Prophet model in the embodiments of the present application. The Prophet model is a time series prediction model that can be used to fit a time series with non-linear trends and can perform trend prediction on a time series with missing values and outliers. According to the input state parameters of the node, the Prophet model can calculate the real-time state evaluation value of the node.

[0170] It can be understood that the above is only described by taking the Prophet model as an example, and in fact, other models can also be used to evaluate the state parameters of multiple nodes. The embodiments of the present application do not make specific limitations on this.

[0171] Step S152: obtaining the priority of the at least one node according to the calculated real-time state evaluation value of the node and the historical state evaluation value of the node.

[0172] It should be understood that the prerequisite for marking the priority of the node after comprehensive evaluation of the real-time state evaluation value of the node and the historical state evaluation value of the node is to meet the preset condition, that is, the evaluation value of the node is greater than the preset threshold, and then the node can enter the node priority list to perform the recovery operation on the data.

[0173] Step S153: determining a node priority list according to the priority of the node.

[0174] Specifically, in the node priority list, the plurality of nodes are arranged in descending order of priority. The node with higher priority receives data to perform recovery operation first. The above-mentioned manner can further improve the efficiency of data disaster recovery.

[0175] Based on the node state evaluation model, the real-time state of the plurality of nodes in the distributed cluster is evaluated, and the node priority list that can be used to recover data or process data is obtained based on sufficient consideration of the cluster difference. For at least one node included in the node priority list, after receiving the data to be recovered, the data can be quickly recovered to further improve the rate of data disaster recovery.

[0176] For the convenience of those skilled in the art, the method of data preprocessing is described below in combination with specific examples in Figure 6 It should be understood that the method in Figure 6 is a refinement of S130 in the data disaster recovery method shown in Figure 2 and is not intended to limit the embodiments of the present application to the specific scenarios illustrated.

[0177] For step S130, the specific operation of preprocessing is not specifically limited in the embodiments of the present application.

[0178] Optionally, in an implementation manner, the data is preprocessed by means of null value processing, data standardization (normalization) processing and category type feature processing. The preprocessing operation includes the following steps:

[0179] Step S131: performing category type feature processing on non-numerical data in the data of the first evaluation data set to obtain data after category type feature processing, wherein the category type feature processing is used to convert non-numerical data into numerical data.

[0180] The specific manner of category type feature processing is not specifically limited in the embodiments of the present application.

[0181] In a possible implementation manner, the data in the first evaluation data set can be processed by category type feature processing by using Categorical. Categorical is a data type in Python software, which can assign numerical values to a limited group of discrete categories, can improve efficient data storage and convenient non-numerical data processing, and at the same time, retains meaningful names for numerical values.

[0182] Exemplarily, the category type features are first screened out by using Python software, and then the category type features in the first evaluation data set are converted into numerical features by using Categorical.

[0183] It should be understood that the above is only described by taking the Categorical as an example for processing the category type features, and the embodiments of the present application are not limited thereto. In fact, other ways can also be used instead of the Categorical.

[0184] Optionally, in an implementation, the category type features of the first evaluation data set can also be processed by any of the following: serial number coding, one-hot coding, and binary coding.

[0185] Step S132: Null value processing is performed on the numerical type data to obtain data after the null value processing, the numerical type data including the original numerical data in the first evaluation data set and the numerical data after the category type feature processing in step S131.

[0186] In other words, assuming that there are data sets of three features (which can be represented by three fields) A, B, and C in the first evaluation data set, the feature A is a non-numerical type feature, and the features B and C are numerical type features, the feature A is first converted from the non-numerical type feature into the numerical type feature, and then the null value processing is performed on the features A, B, and C to obtain the data after the null value processing.

[0187] The specific manner of the null value processing is not specifically limited in the embodiments of the present application.

[0188] In a possible implementation, the data rows containing the null values can be screened out by using Python software, and the Lagrange interpolation method is used to perform interpolation processing on the null values of the data set, and the interpolation function is as follows:

[0189] wherein,

[0190] wherein, n represents the total number of rows of the data set, x represents the position of the null value, x i , x j represents the value (or position) of the independent variable, ω i represents the barycentric weight, and y i represents the value when the independent variable (or position) is x i .

[0191] It should be understood that the above is only described by taking the Lagrange interpolation method as an example for processing the category type features, and the embodiments of the present application are not limited thereto. In fact, other ways can also be used instead of the Lagrange interpolation method.

[0192] Optionally, in an implementation, the data set null values can also be interpolated by Newton interpolation, nearest neighbor algorithm (KNN interpolation) and improved methods thereof. In this regard, the embodiments of the present application are not specifically limited.

[0193] Step S133: performing data normalization processing on the data after the null value processing to obtain the pre-processed data.

[0194] The specific manner of data normalization processing is not specifically limited by the embodiments of the present application.

[0195] In a possible implementation, the data after the null value processing can be normalized by z-score standardization in Python software, and the standardization function is as follows:

[0196]

[0197] Wherein, u represents the mean of the data after the null value processing, σ represents the standard deviation of the data after the null value processing, z represents the result of normalization processing, and x represents the data after the null value processing, i.e. the data before normalization processing.

[0198] It should be understood that the above is only described by taking z-score standardization as an example for category type feature processing, and the embodiments of the present application are not limited thereto. In fact, other methods can also be used instead of z-score standardization.

[0199] Optionally, in an implementation, the data after the null value processing can also be normalized by min-max standardization, Sigmoid function, etc. In this regard, the embodiments of the present application are not specifically limited.

[0200] Wherein, the data normalization processing on the data after the null value processing can avoid the influence of the dimension between data sample features and overfitting problems.

[0201] It should be understood that the pre-processing operation on the first evaluation data set in Figure 6 is only to facilitate the understanding of those skilled in the art Figure 2 S130 in the data disaster recovery method shown in the figure, and the pre-processing operation is not limited to only being applicable to the first evaluation data set of the embodiments of the present application. Similarly, Figure 6 the pre-processing operation in is also applicable to the second evaluation data set in the embodiments of the present application. In this regard, the embodiments of the present application will not be described again for the sake of brevity.

[0202] To facilitate the understanding of those skilled in the art, the data disaster recovery method provided by the present application is described below in combination with specific examples in Figure 7 It should be understood that the pre-processing operation on the first evaluation data set in Figure 7The examples in the foregoing merely facilitate the understanding of the data disaster recovery method provided by the embodiments of the present application by the person skilled in the art, and are not intended to limit the embodiments of the present application to the specific scenarios exemplified. The person skilled in the art can make various equivalent modifications or changes according to the examples in the foregoing, and such modifications or changes also fall within the scope of the embodiments of the present application. Figure 7 Obviously, various equivalent modifications or changes can be made to the examples in the foregoing, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0203] Figure 7 is an example flowchart of the data disaster recovery method provided by the embodiments of the present application. It can be understood that, as an example but not limitation, Figure 7 The method in the foregoing can be applied to the application scenarios shown in Figure 1 . Figure 7 The related terms or explanations involved in the foregoing can refer to the foregoing description, and will not be repeated here. As shown in Figure 7 , the method comprises the following steps:

[0204] S710: Node failure warning.

[0205] Exemplarily, after the user configures the open source big data monitoring platform to monitor the heterogeneous big data distributed cluster system, the user can monitor the node failure in the distributed cluster system. When the node failure in the distributed cluster system is monitored, S720 is performed.

[0206] S720: The node determines whether the failure type is a data node failure.

[0207] S730: When the failure type is a data node failure type, a first evaluation data set and a second evaluation data set are obtained.

[0208] S740: The first evaluation data set is preprocessed to obtain a preprocessed first evaluation data set, and the second evaluation data set is preprocessed to obtain a preprocessed second evaluation data set.

[0209] The process of data preprocessing can refer to the description in the foregoing steps S130 and Figure 6 , and will not be repeated here for brevity.

[0210] S750: According to the business sensitivity evaluation model and the preprocessed first evaluation data set, a data recovery priority list is determined, and according to the node state evaluation model and the preprocessed second evaluation data set, a node recovery priority list is determined.

[0211] The process of determining the data recovery priority list can refer to the description in the foregoing steps S140 and Figure 4 , and will not be repeated here for brevity.

[0212] The process of determining the node recovery priority list can refer to the description in the foregoing step S150 and Figure 5 For brevity, details are not repeated here.

[0213] S760: determining a fault handling strategy according to the data recovery priority list and the node recovery priority list.

[0214] S770: executing the fault handling strategy.

[0215] If the fault type is not the data node fault type, i.e., the fault type is the name node fault, the corresponding data of the node will not be acquired, and a fault handling strategy is directly made. The fault handling strategy directly makes the name node exit the safe mode. The fault handling strategy is not equal to the fault handling strategy in S760.

[0216] It can be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0217] A data disaster recovery method is provided, Figure 8 A structure block diagram of a data disaster recovery device provided by the embodiments of the present application is shown, and only parts related to the embodiments of the present application are shown for brevity.

[0218] Please refer to Figure 8 The data disaster recovery device 800 is applied to a distributed cluster system including a plurality of nodes, and the device 800 includes a determination module 810, an acquisition module 820, a preprocessing module 830, and an execution module 840.

[0219] In some possible implementation manners, the determination module 810 is configured to determine a fault type and a range of to-be-recovered data when a node in the distributed cluster system fails.

[0220] In some possible implementation manners, the acquisition module 820 is configured to acquire a first evaluation data set and a second evaluation data set when the fault type is the data node fault type, where the first evaluation data set includes a business sensitivity feature and / or a data block access feature of the to-be-recovered data, and the second evaluation data set includes a state parameter of a node other than a fault data node in the plurality of nodes.

[0221] In some possible implementation manners, the preprocessing module 830 is configured to perform preprocessing on the first evaluation data set obtained by the obtaining module 820 to obtain a preprocessed first evaluation data set, and perform preprocessing on the second evaluation data set to obtain a preprocessed second evaluation data set.

[0222] Optionally, the preprocessing operation includes:

[0223] category type feature processing, null value processing, and data normalization processing, where the category type feature processing is configured to convert non-numerical data into numerical data.

[0224] In some possible implementation manners, the determining module 810 is further configured to determine a data recovery priority list according to the preprocessed first evaluation data set and a business sensitivity evaluation model, where the business sensitivity evaluation model is configured to output a priority of to-be-recovered data according to the preprocessed first evaluation data set, and the data recovery priority list is configured to represent an order of the priority of the to-be-recovered data.

[0225] Optionally, the business sensitivity feature of the to-be-recovered data includes one or more of the following features: a priority feature of a business system to which the data block belongs, a priority feature of a function module to which the data block belongs, a business attribute feature of the data block, and a timeliness feature of the data block.

[0226] Optionally, the business sensitivity evaluation model is configured to output the priority of the to-be-recovered data according to the preprocessed first evaluation data set, including:

[0227] The business sensitivity evaluation model is configured to output the priority of the to-be-recovered data according to a hotness value feature of the to-be-recovered data and the business sensitivity feature of the data block.

[0228] The hotness value feature of the to-be-recovered data is obtained by analyzing a data block access feature based on a neural network algorithm.

[0229] Optionally, the data block access feature is determined according to one or more of the following parameters: a file name, a file operation type, a file operation time, and a file operation permission.

[0230] In some possible implementation manners, the determining module 810 is further configured to determine a node priority list according to the preprocessed second evaluation data set and a node state evaluation model, where the node state evaluation model is configured to evaluate real-time states of the nodes according to the preprocessed second evaluation data set and output the node priority list, the node priority list includes at least one node, the at least one node is a node whose state meets a preset condition in the plurality of nodes, and the node priority list is configured to represent a priority order of the at least one node.

[0231] Optionally, the node state evaluation model is configured to evaluate real-time states of the nodes according to the preprocessed second evaluation data set and output the node priority list, including:

[0232] The node state evaluation model is configured to determine the node priority list according to state parameters of the nodes and historical state evaluation values of the nodes.

[0233] The state parameters include one or more of the following parameters: CPU utilization, memory usage, disk I / O usage, network bandwidth usage, command response time, command queue length, and disk usage.

[0234] In some possible implementation manners, the determining module 810 is further configured to determine a fault processing strategy according to the data recovery priority list and the node priority list, where the fault processing strategy is that the to-be-recovered data is sequentially sent to the at least one node according to the data recovery priority list.

[0235] In some possible implementation manners, the executing module 840 is further configured to execute the fault processing strategy.

[0236] Optionally, the executing the fault processing strategy includes:

[0237] The to-be-recovered data is sequentially sent to the at least one node for processing according to the priority order of the to-be-recovered data and the priority order of the at least one node.

[0238] Optionally, after the to-be-recovered data is sequentially sent to the at least one node for processing according to the priority order of the to-be-recovered data and the priority order of the at least one node, the method further includes:

[0239] If there is remaining data in the to-be-recovered data, the remaining data is sent to a node that has completed data processing in the at least one node, where the remaining data refers to data in the to-be-recovered data that has not been processed by the at least one node.

[0240] It should be understood that the data disaster recovery apparatus according to the embodiments of the present application can correspond to the foregoing method embodiments, and the foregoing and other management operations and / or functions of each module in the data disaster recovery apparatus are respectively for implementing the corresponding steps of the methods in the foregoing method embodiments, and thus can also achieve the beneficial effects in the foregoing method embodiments, which will not be described here for brevity.

[0241] Figure 9 A structural diagram of data disaster recovery provided by the embodiments of the present application is shown in FIG. 9. As shown in the figure, the apparatus 900 includes at least one processor 90 (only one processor is shown in the figure), a memory 91, and a computer program 92 stored in the memory 91 and executable on the at least one processor 90, wherein the processor 90 is configured to implement the steps in any of the foregoing method embodiments for data disaster recovery (such as the method in the foregoing Figure 9 Figure 9 Figure 2 Figure 7

[0242] The apparatus 900 can be a desktop computer, a notebook, a palm computer, a cloud server, and other computing devices. The data disaster recovery apparatus can include, but is not limited to, the processor 90 and the memory 91. Those skilled in the art can understand that the apparatus 900 is only an example of the data disaster recovery apparatus, and does not constitute a limitation on the apparatus 900, and can include more or fewer components than those shown in the figure, or combine certain components, or different components, for example, can also include input / output devices, network access devices, etc. Figure 9

[0243] The processor 90 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0244] ​​​​​The memory 91 may, in some embodiments, be an internal storage unit of the device 900, such as a hard disk or a memory of the device 900 for data disaster recovery. The memory 91 may, in other embodiments, also be an external storage device of the device 900, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like equipped on the device 900. Further, the memory 91 may also include both an internal storage unit and an external storage device of the device 900. The memory 91 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of the computer program, and the like. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0245] It should be noted that the information interaction and execution process between the above devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought by the above devices / units can be referred to the method embodiments part, which will not be described here.

[0246] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0247] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0248] The embodiments of the present application provide a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal is caused to implement the steps in each of the above method embodiments.

[0249] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned embodiment methods through a computer program to instruct relevant hardware to complete, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can at least include any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0250] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0251] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0252] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are only schematic. The division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0253] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0254] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data disaster recovery method, characterized in that, The method is applied to a distributed cluster system, the distributed cluster system comprising multiple nodes, the method comprising: When a node in the distributed cluster system fails, the type of failure and the scope of data to be recovered are determined. When the fault type is a data node fault type, a first evaluation dataset and a second evaluation dataset are obtained, wherein the first evaluation dataset includes the business sensitivity characteristics and / or data block access characteristics of the data to be recovered, and the second evaluation dataset includes the status parameters of the nodes other than the faulty data node among the plurality of nodes. The first evaluation dataset is preprocessed to obtain a preprocessed first evaluation dataset, and the second evaluation dataset is preprocessed to obtain a preprocessed second evaluation dataset. Based on the preprocessed first evaluation dataset and the business sensitivity assessment model, a data recovery priority list is determined. The business sensitivity assessment model outputs the priority of the data to be recovered based on the preprocessed first evaluation dataset. The data recovery priority list represents the priority order of the data to be recovered. The business sensitivity assessment model outputs the priority of the data to be recovered based on the preprocessed first evaluation dataset, including: the business sensitivity assessment model outputs the priority of the data to be recovered based on the popularity value characteristics and the business sensitivity characteristics of the data to be recovered; the popularity value characteristics of the data to be recovered are obtained by analyzing data block access characteristics based on a neural network algorithm. Based on the preprocessed second evaluation dataset and the node state evaluation model, a node priority list is determined. The node state evaluation model evaluates the real-time state of each node based on the preprocessed second evaluation dataset and outputs the node priority list. The node priority list includes at least one node, which is a node among the plurality of nodes whose state meets preset conditions. The node priority list represents the priority order of the at least one node. The node state evaluation model evaluates the real-time state of each node based on the preprocessed second evaluation dataset and outputs the node priority list, including: the node state evaluation model determines the node priority list based on the state parameters of each node and the historical state evaluation values ​​of each node; wherein the state parameters include one or more of the following parameters: CPU utilization, memory utilization, disk I / O utilization, network bandwidth utilization, command response time, command queue length, and disk utilization. Based on the data recovery priority list and the node priority list, a fault handling strategy is determined. The fault handling strategy refers to sending the data to be recovered to the at least one node in sequence according to the data recovery priority list. Execute the aforementioned fault handling strategy.

2. The method according to claim 1, characterized in that, The business sensitivity characteristics of the data to be recovered include one or more of the following characteristics: the priority characteristics of the business system to which the data block belongs, the priority characteristics of the functional module to which the data block belongs, the business attribute characteristics of the data block, and the timeliness characteristics of the data block.

3. The method according to claim 1, characterized in that, The data block access characteristics are determined based on one or more of the following parameters: file name, file operation type, file operation time, and file operation permissions.

4. The method according to claim 1, characterized in that, The execution of the fault handling strategy includes: The data to be recovered is sent to the at least one node for processing in sequence according to the priority order of the data to be recovered and the priority order of the at least one node.

5. The method according to claim 4, characterized in that, After sending the data to be recovered to the at least one node for processing according to the priority order of the data to be recovered and the priority order of the at least one node, the method further includes: If there is still remaining data in the data to be recovered, the remaining data is sent to the node among the at least one node that has completed data processing, wherein the remaining data refers to the data in the data to be recovered that has not been processed by the at least one node.

6. The method according to claim 1, characterized in that, The preprocessing operation includes one or more of the following processes: Categorical feature processing, null value processing, and data normalization processing are included, wherein the categorical feature processing is used to convert non-numerical data into numerical data.

7. A data disaster recovery device, characterized in that, The device is applied to a distributed cluster system, the distributed cluster system comprising multiple nodes, and the device includes: The determination module is used to determine the fault type and the range of data to be recovered when a node in the distributed cluster system is determined to have failed. The acquisition module is used to acquire a first evaluation dataset and a second evaluation dataset when the fault type is a data node fault type. The first evaluation dataset includes the business sensitivity characteristics and / or data block access characteristics of the data to be recovered, and the second evaluation dataset includes the status parameters of the nodes other than the faulty data node among the plurality of nodes. The preprocessing module is used to preprocess the first evaluation dataset to obtain a preprocessed first evaluation dataset, and to preprocess the second evaluation dataset to obtain a preprocessed second evaluation dataset. The determining module is further configured to determine a data recovery priority list based on the preprocessed first evaluation dataset and the business sensitivity evaluation model, wherein the business sensitivity evaluation model is configured to output the priority of the data to be recovered based on the preprocessed first evaluation dataset, the data recovery priority list is configured to characterize the priority order of the data to be recovered, and the business sensitivity evaluation model is configured to output the priority of the data to be recovered based on the preprocessed first evaluation dataset, including: the business sensitivity evaluation model is configured to output the priority of the data to be recovered based on the popularity value characteristics of the data to be recovered and the business sensitivity characteristics of the data to be recovered; the popularity value characteristics of the data to be recovered are obtained by analyzing the data block access characteristics based on a neural network algorithm; The determining module is further configured to determine a node priority list based on the preprocessed second evaluation dataset and the node state evaluation model. The node state evaluation model is configured to evaluate the real-time state of each node based on the preprocessed second evaluation dataset and output the node priority list. The node priority list includes at least one node, which is a node among the plurality of nodes whose state meets preset conditions. The node priority list is used to characterize the priority order of the at least one node. The node state evaluation model is configured to evaluate the real-time state of each node based on the preprocessed second evaluation dataset and output the node priority list, including: the node state evaluation model is configured to determine the node priority list based on the state parameters of each node and the historical state evaluation values ​​of each node; wherein the state parameters include one or more of the following parameters: CPU utilization, memory utilization, disk I / O utilization, network bandwidth utilization, command response time, command queue length, and disk utilization. The determining module is further configured to determine a fault handling strategy based on the data recovery priority list and the node priority list, wherein the fault handling strategy refers to: sending the data to be recovered to the at least one node in sequence according to the data recovery priority list; The execution module is used to execute the fault handling strategy.

8. A data disaster recovery device, characterized in that, The device includes a processor and a memory coupled together, the memory being used to store a computer program that, when executed by the processor, causes the device to perform the method of any one of claims 1 to 6.

Images