Data processing method, device, apparatus and program product

Abstract

The present invention provides a data processing method, device, equipment and program product, which can be applied to the field of distributed database technology. The method includes: obtaining an execution result set of a query statement, each execution result in the execution result set includes a query statement, a distributed node for executing the query statement and query statement execution completion information; when it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, determining the target distributed node corresponding to the execution result from the execution result set; when the number of task executions of the target distributed node is greater than a preset target threshold, determining the target distributed node as an error node; when it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster, taking the error node offline.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and more specifically to a data processing method, device, equipment and program product. Background Art

[0002] Due to the instability of the physical machine hardware and occasional independent disk redundant array failures, a node was physically damaged, causing subsequent query statements executed on this node to keep reporting errors due to the node hardware problem. The query statements executed on this node kept reporting errors due to the hardware problem of this node, which concealed some other problems and also generated a large number of repeated error messages in the logs, which was not conducive to problem troubleshooting. Summary of the invention

[0003] In view of the above problems, the present invention provides a data processing method, apparatus, device, medium and program product.

[0004] According to a first aspect of the present invention, a data processing method is provided, comprising: obtaining an execution result set of a query statement, wherein the query statement satisfies a statement execution time greater than a preset threshold, and each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information; in a case where it is determined that the query statement execution completion information in the execution result set meets a predetermined completion type, determining a target distributed node corresponding to the execution result from the execution result set; in a case where the number of task executions of the target distributed node is greater than a preset target threshold, determining the target distributed node as an error node, wherein the number of task executions represents the number of execution results whose query statement execution completion information meets the predetermined completion type; in a case where it is determined that there are backup shards corresponding to the shards included in the error node in the distributed cluster, taking the error node offline, wherein the backup shards are distributed on other nodes of the distributed cluster except the error node.

[0005] Optionally, the data processing method also includes: determining the shard information of at least one shard in the faulty node; based on the shard information, querying at least one backup shard corresponding to at least one shard from other nodes in the distributed cluster; when it is determined that there is at least one backup shard on other nodes in the distributed cluster, determining that there is a backup shard corresponding to the shard included in the faulty node in the distributed cluster.

[0006] Optionally, before taking the faulty node offline, the data processing method also includes: performing predetermined task processing on other nodes corresponding to the backup shard to obtain task processing results; based on the task processing results of other nodes corresponding to the backup shard, determining the available status information of the backup shard, so as to take the faulty node offline when the available status information of the backup shard indicates that the backup shard is available.

[0007] Optionally, the data processing method further includes: determining the number of occurrences of the node identifier of the target distributed node from the execution result set; and determining the number of task executions based on the number of occurrences of the node identifier.

[0008] Optionally, the data processing method further includes: when it is determined that no backup shards exist on other nodes of the distributed cluster, storing the situation in a log.

[0009] Optionally, determining a target distributed node corresponding to the execution result from the execution result set includes: determining an initial distributed node corresponding to the execution result from the execution result set; and when it is determined based on the state parameters of the initial distributed node that the network operation of the initial distributed node is normal, using the initial distributed node as the target distributed node.

[0010] A second aspect of the present invention provides a data processing device, comprising:

[0011] An acquisition module is used to obtain an execution result set of a query statement, wherein the query statement satisfies a statement execution time greater than a preset threshold, and each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information; a determination module is used to determine a target distributed node corresponding to the execution result from the execution result set when it is determined that the query statement execution completion information in the execution result set meets a predetermined completion type; an error node determination module is used to determine a target distributed node as an error node when the number of task executions of the target distributed node is greater than a preset target threshold, wherein the number of task executions represents the number of execution results whose query statement execution completion information meets a predetermined completion type; an offline module is used to take the error node offline when it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster, wherein the backup shard is distributed on other nodes of the distributed cluster except the error node.

[0012] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0013] A fourth aspect of the present invention further provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0014] According to an embodiment of the present invention, by obtaining an execution result set of a query statement, each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information. When it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, the target distributed node corresponding to the execution result is determined from the execution result set. When the number of tasks executed by the target distributed node is greater than a preset target threshold, the target distributed node is determined as an error node. When it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster, the error node is offline. When it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster, the error node is offline, thereby solving a series of problems caused by node damage while ensuring the normal execution of the query statement. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0016] Figure 1 An application scenario diagram of a data processing method according to an embodiment of the present invention is shown;

[0017] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown;

[0018] Figure 3 A flowchart of determining backup shards according to an embodiment of the present invention is shown;

[0019] Figure 4 A schematic diagram of sharding in a distributed cluster according to an embodiment of the present invention is shown;

[0020] Figure 5 A structural block diagram of a data processing device according to an embodiment of the present invention is shown; and

[0021] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0025] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] Due to the instability of the physical machine hardware and occasional independent disk redundant array failures, a node may be physically damaged. The inventors have found that due to the damage to the physical node, subsequent query statements executed on this node will continue to report errors due to the node hardware problem, thereby covering up the real reasons for some query statement errors. At the same time, a large number of repeated error messages will be generated in the log, which is not conducive to the troubleshooting of query statement error information.

[0027] In view of this, the present invention provides a data processing method, device, electronic device and medium, the method comprising: obtaining an execution result set of a query statement, each execution result in the execution result set including a query statement, a distributed node for executing the query statement and query statement execution completion information; determining a target distributed node corresponding to the execution result from the execution result set when it is determined that the query statement execution completion information in the execution result set meets a predetermined completion type; determining the target distributed node as an error node when the number of task executions of the target distributed node is greater than a preset target threshold; and taking the error node offline when it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster. The present invention also provides a data processing device, equipment, storage medium and program product.

[0028] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] Figure 1 An application scenario diagram of a data processing method according to an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The data processing method of the disclosed embodiment is described in detail.

[0037] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown.

[0038] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S240.

[0039] In operation S210, a set of execution results of the query statement is obtained.

[0040] In operation S220, when it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, a target distributed node corresponding to the execution result is determined from the execution result set.

[0041] In operation S230 , when the number of executed tasks of the target distributed node is greater than a preset target threshold, the target distributed node is determined as an error node.

[0042] In operation S240, when it is determined that there are backup shards corresponding to the shards included in the faulty node in the distributed cluster, the faulty node is taken offline.

[0043] Optionally, query statement execution status information within a period of time is read from a database table to obtain a query statement execution result set. The query statement satisfies that the statement execution time is greater than a preset threshold. Each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information. The execution result set can also include a task number, a query statement, a node address for executing the query statement, a query statement start execution time, a query statement consumption time, and an error message, etc., as shown in Table 1.

[0044] Table 1 Execution result set display

[0045]

[0046] Optionally, after obtaining the execution result set, it is necessary to process it, count all result sets in which query statements in the execution result have errors, and further clean it to contain only result sets of predetermined completion types.

[0047] Optionally, after obtaining a result set containing only predetermined completion types, the Internet Protocol (IP) address of the error node can be obtained from the predetermined completion type and added to the end of each query statement, and the number of execution results of the predetermined completion type can be counted.

[0048] Optionally, when it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, the target distributed node corresponding to the execution result can be determined from the execution result set. A database table can be created to store the predetermined completion type, and some error messages that are frequently encountered or expected to be detected can be manually inserted. The predetermined completion type may include service exceptions (Lost connection to GBase server), active kill statements (The query is killed by user), disk full (writing file \'. / gctmpdb / metadata / _tmp_\'), etc., but is not limited to this, and the embodiments of the present invention do not limit this. The query statement execution completion information may correspond to the target error information of the query statement in Table 1 for this execution, and the predetermined completion type includes the target error information.

[0049] Optionally, the number of task executions of the target distributed node can be counted, and when the number of task executions of the target distributed node is greater than a preset target threshold, the target distributed node is determined as an error node. The number of task executions can represent the number of execution results whose query statement execution completion information meets the predetermined completion type.

[0050] Optionally, the preset target threshold can be stored in a configuration file, and the configuration file can also include a user module, a statistical strategy module and an exception handling module. The user module can include an email sender mailbox, an email service and a receiving mailbox, and the receiving mailbox can include multiple. The statistical strategy module can include a polling time interval, a query statement statistical time period and a preset target threshold of a predetermined completion type that triggers an alarm strategy, etc. The polling time interval can indicate that polling the database table starts at a certain interval to obtain an execution result set. The exception handling module includes an offline parameter and an email sending parameter. When the offline parameter is set to 0, it can indicate that the error node is not offline. When the offline parameter is set to 1, it can indicate that the error node is offline. When the email sending parameter is set to 0, it can indicate that an error node occurs when no email notification is sent. When the email sending parameter is set to 1, it can indicate that an error node occurs when an email notification is sent.

[0051] Optionally, the polling interval is set to periodically check the nodes to avoid wasting resources due to node errors.

[0052] Optionally, the target execution strategy can be determined based on the exception handling module in the configuration file, and the strategy to be executed can be determined based on the offline setting parameters and email sending parameters. Currently, it supports sending emails, offline error nodes, or executing the above two strategies at the same time. If no strategy is adopted, the execution result of this execution will be recorded in the log, recording the execution completion information of the scheduled completion type, the number of tasks executed of the scheduled completion type, and the proportion of the execution result set; if the strategy is adopted, in addition to recording the execution completion information of the scheduled completion type, the number of tasks executed of the scheduled completion type, and the proportion of the execution result set in this execution, the log also records the strategy execution status and strategy processing results.

[0053] Optionally, the backup shards are distributed on other nodes of the distributed cluster except the failed node. When the failed node fails, the shards of the node cannot be used normally. At this time, it is necessary to find the backup shards on other nodes.

[0054] Optionally, the failure status can indicate a cluster node failure, which is equivalent to being offline. At this time, the Data Definition Language (DDL) and Data Manipulation Language (DML) will not be sent to the node, but will be directly logged. When the node failure is resolved, the node can be set to normal, which is equivalent to the node being back online. At this time, the previously recorded logs will be restored, and the newly initiated DDL and DML will be sent to the node again.

[0055] Optionally, by obtaining an execution result set of a query statement, each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information. When it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, the target distributed node corresponding to the execution result is determined from the execution result set. When the number of tasks executed by the target distributed node is greater than a preset target threshold, the target distributed node is determined as an error node. When it is determined that there are backup shards corresponding to the shards included in the error node in the distributed cluster, the error node is taken offline. When it is determined that there are backup shards corresponding to the shards included in the error node in the distributed cluster, the error node is taken offline, thereby solving a series of problems caused by node damage while ensuring the normal execution of the query statement. The present invention can be used in a scenario where an irreparable fault occurs in a node in a distributed cluster and does not affect other nodes.

[0056] Figure 3 A flow chart for determining backup shards according to an embodiment of the present invention is shown.

[0057] like Figure 3 As shown, the data processing method of this embodiment includes operations S310 to S330.

[0058] In operation S310, shard information of at least one shard in the failed node is determined.

[0059] In operation S320, based on the shard information, at least one backup shard corresponding to at least one shard is queried from other nodes in the distributed cluster.

[0060] In operation S330, when it is determined that at least one backup shard exists on other nodes of the distributed cluster, it is determined that a backup shard corresponding to the shard included in the failed node exists in the distributed cluster.

[0061] Optionally, the shard information of at least one shard in the failed node can be determined, and based on the shard information, at least one backup shard corresponding to at least one shard can be queried from other nodes in the distributed cluster. When it is determined that at least one backup shard exists on other nodes in the distributed cluster, it can be determined that a backup shard corresponding to the shard included in the failed node exists in the distributed cluster. For example, there are 3 nodes, namely node 1, node 2 and node 3. Node 1 has 2 shards, namely n1 and n2, node 2 has 2 shards, namely n2 and n3, and node 3 has 2 shards, namely n3 and n1. Node 2 is an failed node. Based on the known shard information of node 2, the backup shards related to node 2 can be found from node 1 and node 3 respectively. On the basis that backup shard n2 exists in node 1 and backup shard n3 exists in node 3, it can be determined that backup shards exist in failed node 2.

[0062] Optionally, before taking the faulty node offline, the data processing method also includes: performing predetermined task processing on other nodes corresponding to the backup shard to obtain task processing results; based on the task processing results of other nodes corresponding to the backup shard, determining the available status information of the backup shard, so as to take the faulty node offline when the available status information of the backup shard indicates that the backup shard is available.

[0063] Optionally, the show nodes statement can be used to perform scheduled task processing on other nodes corresponding to the backup shard to obtain task processing results, where the task processing results are shown in Table 2. Based on the task processing results of other nodes corresponding to the backup shard, the available status information of the backup shard is determined, so that the faulty node can be taken offline when the backup shard is available.

[0064] Table 2 Task processing results

[0065]

[0066] Optionally, the data processing method further includes: determining the number of occurrences of the node identifier of the target distributed node from the execution result set; and determining the number of task executions based on the number of occurrences of the node identifier.

[0067] Optionally, the number of node identifiers of the target distributed node can be determined from the execution result set, and the number of task executions can be determined based on the number of node identifiers. For example, if the target distributed node is determined to be node 2, the node identifier is an IP address associated with node 2, and if it is determined that the IP address associated with node 2 appears 10 times, then the number of task executions can be determined to be 10.

[0068] Optionally, determining the number of tasks to be executed based on the number of node identifier occurrences can help reduce unnecessary overhead, thereby improving cost-effectiveness and overall execution efficiency.

[0069] Optionally, the data processing method also includes: when it is determined that there are no backup shards on other nodes of the distributed cluster, storing this situation in a log and recording the erroneous node that cannot be taken offline.

[0070] Optionally, if it is determined that there are no backup shards on other nodes of the distributed cluster, an email is sent to notify the database of the abnormal situation and inform that the faulty node cannot be offline. If it is determined that there are backup shards on other nodes of the distributed cluster, the faulty node can be offline and an email is sent to notify the database of the abnormal situation and the offline of the faulty node.

[0071] Optionally, determining a target distributed node corresponding to the execution result from the execution result set includes: determining an initial distributed node corresponding to the execution result from the execution result set; and when it is determined based on the state parameters of the initial distributed node that the network operation of the initial distributed node is normal, using the initial distributed node as the target distributed node.

[0072] Optionally, the initial distributed node corresponding to the execution result can be determined from the execution result set. For example, in the execution result set, the execution result involves three nodes, namely node 1, node 2 and node 3, then these three nodes are determined as the initial distributed nodes. When it is determined based on the state parameters of the initial distributed node that the initial distributed node is in a normal network operation, the initial distributed node is used as the target distributed node. The state parameters may include a status field and a datastate field. The status field may represent the state of the process, such as running, paused, waiting or terminated state, etc. The datastate field may represent the current valid state of the data. When the status field represents online and the datastate field has no abnormalities, it is determined that the initial distributed node is in a normal network operation state, and the initial distributed node may be used as the target distributed node.

[0073] Optionally, whether the initial distributed node is in a normal network operation state can be determined based on the state parameters of the initial distributed node, thereby eliminating the situation where a node with abnormal network operation is mistakenly identified as an erroneous node, thereby improving the accuracy of troubleshooting.

[0074] Optionally, taking the faulty node offline may include updating the running flag of the shard of the faulty node to an offline flag. The command used for the faulty node is gcadmin setnodestate ip <state>, where state is set to failed. If you need to restore the node, you need to manually set the node state to normal.

[0075] Figure 4 A schematic diagram of sharding in a distributed cluster according to an embodiment of the present invention is shown.

[0076] like Figure 4 As shown, three nodes are configured in the distributed cluster, namely gnode1 410, gnode2 420 and gnode3 430. Gnode1 has n1 and n2 shards, gnode2 has n2 and n3 shards, and gnode3 has n1 and n3 shards.

[0077] Optionally, you can first use the Test Processing Unit-H (TPC-H) model, which is a benchmark model for evaluating online analytical processing performance. It simulates the transaction behavior between suppliers and buyers. The TPC-H model is used to simulate 22 queries of the commodity retail decision support system, involving 22 complex query flow statements and 2 update flow statements with insert and delete program segments. The query statements cover complex operations such as statistical grouping, sorting, aggregation operations, subqueries, and multi-table associations.

[0078] Optionally, set the statement execution time to 3s. When the execution time of the query statement is greater than 3s, a system table will be automatically created. The system table is used to store the execution result set of the query statement with an execution time greater than 3s. Create a database table to store the scheduled completion types, such as service exceptions, active kill statements, full disk, etc. You can set the data in the configuration file. For example, poll the system table once every half an hour to obtain the execution result set within half an hour, and set the preset target threshold of the number of task executions of the target distributed node to 10. The offline parameter in the exception handling module is set to 1, which means that when the number of task executions of the scheduled completion type is greater than the preset target threshold and there are available shards, the error node will be offline. The email sending parameter is set to 1, which means that when the number of task executions of the scheduled completion type is greater than the preset target threshold, an email will be sent to inform the management personnel.

[0079] Optionally, the single machine of the gnode2 node is set as an error node, and the query statements related to the TPC-H model are cyclically executed on the nodes of the distributed cluster. Obtain a set of execution results within half an hour, and when it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, determine the target distributed node corresponding to the execution result from the execution result set. The number of task executions on the target distributed nodes can be counted separately, and the number of task executions represents the number of execution results whose query statement execution completion information meets the predetermined completion type. Statistics show that the number of task executions of gnode1 is 7, the number of task executions of gnode2 is 15, and the number of task executions of gnode3 is 8. The number of task executions of gnode2 is greater than the preset target threshold of 10, thereby, the target distributed node can be determined as an error node, and the single machine service of gnode2 can be determined to be abnormal. When it is determined that the n2 and n3 shards of gnode2 have backup shards in gnode1 and gnode3 respectively, it is necessary to determine the task processing results of gnode1 and gnode3. The scheduled task processing is performed on the other nodes gnode1 and gnode3 corresponding to the backup shards to obtain the task processing results. If the task processing results indicate that the backup shards are normal, it can be determined that the backup shards are available. If the backup shards gnode1 and gnode3 exist and are available, the faulty node gnode2 can be taken offline to ensure the normal execution of the query statement.

[0080] Based on the above data processing method, the present invention also provides a data processing device. Figure 5 The device is described in detail.

[0081] Figure 5 A structural block diagram of a data processing device according to an embodiment of the present invention is shown.

[0082] like Figure 5 As shown, the data processing device 500 of this embodiment includes an acquisition module 510 , a determination module 520 , an error node determination module 530 and an offline module 540 .

[0083] The acquisition module 510 is used to obtain an execution result set of a query statement, wherein the query statement satisfies that the statement execution time is greater than a preset threshold, and each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information.

[0084] The determination module 520 is configured to determine a target distributed node corresponding to the execution result from the execution result set when it is determined that the query statement execution completion information in the execution result set meets a predetermined completion type.

[0085] The error node determination module 530 is used to determine the target distributed node as an error node when the number of task executions of the target distributed node is greater than a preset target threshold, wherein the number of task executions represents the number of execution results of the query statement execution completion information that meets the predetermined completion type.

[0086] The offline module 540 is used to take the faulty node offline when it is determined that there are backup shards corresponding to the shards included in the faulty node in the distributed cluster, wherein the backup shards are distributed on other nodes of the distributed cluster except the faulty node.

[0087] Optionally, by obtaining an execution result set of a query statement, each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information, and in the case where it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, the target distributed node corresponding to the execution result is determined from the execution result set, and in the case where the number of tasks executed by the target distributed node is greater than a preset target threshold, the target distributed node is determined as an error node, and in the case where it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster, the error node is taken offline. In the case where it is determined that there is a backup shard corresponding to the shard included in the error node in the distributed cluster, the error node is taken offline, thereby solving a series of problems caused by node damage while ensuring the normal execution of the query statement.

[0088] Optionally, the data processing device 500 of this embodiment further includes a shard information determination module, a query module and a backup shard determination module.

[0089] The shard information determination module is used to determine the shard information of at least one shard in the error node.

[0090] The query module is used to query at least one backup shard corresponding to at least one shard from other nodes of the distributed cluster based on the shard information.

[0091] The backup shard determination module is used to determine the backup shard corresponding to the shard included in the failed node in the distributed cluster when it is determined that there is at least one backup shard on other nodes of the distributed cluster.

[0092] Optionally, the data processing device 500 of this embodiment further includes a task processing result module and an available status information module.

[0093] The task processing result module is used to perform scheduled task processing on other nodes corresponding to the backup shards to obtain task processing results.

[0094] The available status information module is used to determine the available status information of the backup shard based on the task processing results of other nodes corresponding to the backup shard, so as to take the faulty node offline when the available status information of the backup shard indicates that the backup shard is available.

[0095] Optionally, the data processing device 500 of this embodiment further includes a node identification occurrence quantity module and a task execution quantity determination module.

[0096] The node identification occurrence number module is used to determine the node identification occurrence number of the target distributed node from the execution result set.

[0097] The task execution quantity determination module is used to determine the task execution quantity according to the number of node identifiers.

[0098] Optionally, the data processing device 500 of this embodiment further includes a sending module.

[0099] The emission module is used to store the situation in the log when it is determined that there is no backup shard on other nodes of the distributed cluster.

[0100] Optionally, the determination module includes: an initial distributed node unit and a target distributed node unit.

[0101] The initial distributed node unit is used to determine the initial distributed node corresponding to the execution result from the execution result set.

[0102] The target distributed node unit is used to use the initial distributed node as the target distributed node when it is determined based on the state parameters of the initial distributed node that the network operation of the initial distributed node is normal.

[0103] Optionally, the offline module includes: an update unit.

[0104] The updating unit is used to update the running flag of the shard of the failed node to an offline flag.

[0105] Optionally, any multiple modules among the acquisition module 510, the determination module 520, the error node determination module 530 and the offline module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the acquisition module 510, the determination module 520, the error node determination module 530 and the offline module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the acquisition module 510, the determination module 520, the error node determination module 530 and the offline module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be executed.

[0106] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown.

[0107] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0108] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0109] Optionally, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0110] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0111] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, optionally, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0112] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present invention.

[0113] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0114] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0115] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. Optionally, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0116] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed completely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0117] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0119] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.< / state>

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining an execution result set of a query statement, wherein the query statement satisfies that the statement execution time is greater than a preset threshold, and each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information; In the case where it is determined that the query statement execution completion information in the execution result set meets the predetermined completion type, determining the target distributed node corresponding to the execution result from the execution result set, the predetermined completion type includes service exception and active termination statement, The step of determining a target distributed node corresponding to the execution result from the execution result set includes: Determining an initial distributed node corresponding to the execution result from the execution result set; In a case where it is determined based on the state parameters of the initial distributed node that the network of the initial distributed node is operating normally, using the initial distributed node as the target distributed node; When the number of task executions of the target distributed node is greater than a preset target threshold, the target distributed node is determined as an error node, wherein the number of task executions represents the number of execution results whose query statement execution completion information conforms to a predetermined completion type; When it is determined that there are backup shards corresponding to the shards included in the faulty node in the distributed cluster, the faulty node is taken offline, and the faulty node represents that the node is damaged, wherein the backup shards are distributed on other nodes of the distributed cluster except the faulty node.

2. The method according to claim 1, characterized in that: The method further comprises: Determine the shard information of at least one shard in the faulty node; Based on the shard information, query at least one backup shard corresponding to the at least one shard from the other nodes of the distributed cluster; In the case where it is determined that the at least one backup shard exists on the other nodes of the distributed cluster, it is determined that the backup shard corresponding to the shard included in the failed node exists in the distributed cluster.

3. The method according to claim 1, characterized in that Before taking the faulty node offline, the method further includes: Performing predetermined task processing on other nodes corresponding to the backup shards to obtain task processing results; Based on the task processing results of other nodes corresponding to the backup shard, the availability status information of the backup shard is determined, so that the faulty node is taken offline when the availability status information of the backup shard indicates that the backup shard is available.

4. The method according to claim 1, characterized in that The method further comprises: Determining the number of occurrences of the node identifier of the target distributed node from the execution result set; The number of tasks executed is determined according to the number of occurrences of the node identifiers.

5. The method according to claim 1, characterized in that The method further comprises: When it is determined that the backup shard does not exist on the other nodes of the distributed cluster, the situation is stored in a log.

6. The method according to claim 1, characterized in that The taking the faulty node offline comprises: The operation flag of the shard of the faulty node is updated to an offline flag.

7. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire an execution result set of a query statement, wherein the query statement satisfies that the statement execution time is greater than a preset threshold, and each execution result in the execution result set includes a query statement, a distributed node for executing the query statement, and query statement execution completion information; a determination module, configured to determine a target distributed node corresponding to the execution result from the execution result set when it is determined that the query statement execution completion information in the execution result set conforms to a predetermined completion type, wherein the predetermined completion type includes a service exception and an active termination statement, An initial distributed node unit, used to determine an initial distributed node corresponding to the execution result from the execution result set; A target distributed node unit, configured to use the initial distributed node as the target distributed node when it is determined based on the state parameters of the initial distributed node that the network operation of the initial distributed node is normal; An error node determination module, used for determining the target distributed node as an error node when the number of task executions of the target distributed node is greater than a preset target threshold, wherein the number of task executions represents the number of execution results of the query statement execution completion information that meets the predetermined completion type; The offline module is used to take the faulty node offline when it is determined that there is a backup shard corresponding to the shard included in the faulty node in the distributed cluster, wherein the faulty node represents that the node is damaged, wherein the backup shard is distributed on other nodes of the distributed cluster except the faulty node.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images