Abnormity processing method and device for big data cluster nodes and electronic equipment

Abstract

The invention discloses a big data cluster node-oriented exception handling method and apparatus, and an electronic device. When a first control module detects that processing of a data processing request currently received by a home terminal cluster is abnormal, an abnormal instruction is sent to an online service module, so that the online service module reconstructs a data processing task based on the data processing request and sends the data processing task to an opposite terminal cluster; receiving the data processing task based on the opposite-end cluster, and processing the data processing task based on the cluster node of the opposite-end cluster; and when the first control module detects that the home terminal cluster and / or the opposite terminal cluster has output a data feedback result corresponding to the data processing request, sending a task stop instruction to the cluster which does not output the data feedback result. The data processing requests are processed through the cluster nodes of the opposite-end cluster and the cluster nodes of the home-end cluster, the processing speed of the data processing requests is increased, and normal response of the clusters to all the data processing requests is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an exception handling method, device and electronic equipment for big data cluster nodes. Background Art

[0002] Distributed big data clusters are typically large, comprised of numerous components, and possess complex inter-component relationships. Over time, as equipment in big data clusters ages, they can experience node failures, component anomalies, and disk read / write failures, making it impossible to respond to data processing requests. Big data clusters typically have a certain level of disaster recovery capability, automatically detecting and recovering from these node anomalies. However, this recovery period is typically lengthy, during which all data processing requests accessing the anomalous cluster node will fail.

[0003] To address the above issue, existing techniques primarily involve switching the entire cluster. Specifically, when a node anomaly is detected in the current cluster, another cluster is enabled to process all requests corresponding to the current cluster. However, switching the entire cluster not only affects the processing time of requests accessing the abnormal cluster node, but also the processing time of requests accessing other normal cluster nodes. Furthermore, after switching to another cluster, the actual physical distance of data transmission increases, thus increasing the processing time of all requests accessing cluster nodes in the other cluster. In other words, switching the entire cluster results in a longer processing time for all data processing requests accessing cluster nodes. Summary of the Invention

[0004] The present invention provides an exception handling method, device and electronic equipment for big data cluster nodes, which improves the processing speed of data processing requests and ensures the normal response of the cluster to all data processing requests.

[0005] According to one aspect of the present invention, a method for handling exceptions in a big data cluster node is provided, the method comprising:

[0006] When the first control module detects that a data processing request currently received by the local cluster is processed abnormally, an abnormal instruction is sent to the online service module, so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster;

[0007] Receive data processing tasks based on the peer cluster and process the data processing tasks based on the cluster nodes of the peer cluster;

[0008] When the first control module detects that the local cluster and / or the opposite cluster has output a data feedback result corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result.

[0009] According to another aspect of the present invention, there is provided an exception handling device for a big data cluster node, the device comprising:

[0010] an abnormal instruction sending module, configured to send an abnormal instruction to the online service module when the first control module detects an abnormality in processing a data processing request currently received by the local cluster, so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster;

[0011] A task processing module is used to receive data processing tasks based on the peer cluster and process the data processing tasks based on the cluster nodes of the peer cluster;

[0012] The task stopping module is configured to send a task stopping instruction to the cluster that has not output the data feedback result when the first control module detects that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the exception handling method for a big data cluster node according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, which are used to enable a processor to implement the exception handling method for a big data cluster node according to any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the exception handling method for a big data cluster node according to any embodiment of the present invention.

[0019] The technical solution of the embodiment of the present invention is to send an exception instruction to the online service module when the first control module detects that the data processing request currently received by the local cluster is processed abnormally, so that the online service module rebuilds the data processing task based on the data processing request and sends the data processing task to the opposite cluster, thereby facilitating the normal processing of the data processing task corresponding to the data processing request by the cluster node of the opposite cluster, ensuring the processing time of the data processing request, and avoiding the problem of the data processing request processing time being too long or being unable to respond to the data processing request. When the opposite cluster receives the data processing task, the data processing task is processed based on the cluster node of the opposite cluster. That is, while the cluster node of the local cluster processes the data processing request, the cluster node of the opposite cluster processes the data processing request, thereby ensuring the processing speed of the data processing request. When the first control module detects that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result, so as to avoid wasting the computing resources of the cluster that has not output the data feedback result, facilitate the cluster that has not output the data feedback result to process other data processing requests, and thus improve the computing resource utilization of the cluster. The present invention solves the problem of increased processing time of all requests caused by overall cluster switching in the prior art, shortens the processing time of data processing requests when the cluster node of the local cluster is abnormal, and avoids the problem of data processing requests being unable to respond due to abnormal cluster nodes. When the cluster node of the local cluster processes the data processing request abnormally, the data processing request is processed by both the cluster node of the opposite cluster and the cluster node of the local cluster, thereby improving the processing speed of the data processing request. Moreover, the above process only affects the data processing request accessing the current cluster node of the local cluster, and does not affect the data processing request accessing other cluster nodes of the local cluster, reducing the impact range of the abnormal cluster node and ensuring the processing speed of data processing requests accessing other cluster nodes of the local cluster.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1This is a flowchart of an exception handling method for a big data cluster node provided by an embodiment of the present invention;

[0023] Figure 2 This is a structural example diagram of an exception handling method for a big data cluster node provided by an embodiment of the present invention;

[0024] Figure 3 This is an example flow chart of request processing anomaly detection provided by an embodiment of the present invention;

[0025] Figure 4 This is an example diagram of a process for recording an exception log according to an embodiment of the present invention;

[0026] Figure 5 This is a flowchart of an exception handling method for a big data cluster node provided by an embodiment of the present invention;

[0027] Figure 6 1 is an example flow chart of a verification data processing request provided by an embodiment of the present invention;

[0028] Figure 7 This is an example diagram of a process for handling an early warning of request exceptions provided by an embodiment of the present invention;

[0029] Figure 8 This is a flowchart of an exception handling method for a big data cluster node provided by an embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of the structure of an exception handling device for a big data cluster node provided by an embodiment of the present invention;

[0031] Figure 10 It is a structural diagram of an electronic device that implements the exception handling method for a big data cluster node according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] Figure 1 This is a flowchart of a method for handling exceptions of a big data cluster node provided by the first embodiment of the present invention. This embodiment is applicable to the case where, when an exception is detected in the processing of a data processing request currently received by the local cluster, the cluster nodes of the local cluster and the opposite cluster handle the data processing request. This method can be executed by an exception handling device of a big data cluster node. The exception handling device of the big data cluster node can be implemented in the form of hardware and / or software. The exception handling device of the big data cluster node can be configured in electronic devices such as mobile phones, computers or servers. Figure 1 As shown, the method includes:

[0036] S110. When the first control module detects an exception in processing a data processing request currently received by the local cluster, an exception instruction is sent to the online service module, so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster.

[0037] The first control module is configured to detect whether a cluster node in the local cluster encounters an anomaly in processing a data processing request. The data processing request may be a data processing request sent by a client to an application. The local cluster is the cluster currently connected to the application and is configured to process the data processing request sent by the client. The local cluster includes multiple cluster nodes configured to process different data processing requests. For example, different cluster nodes in the local cluster may process data processing requests of different service types.

[0038] A data processing request exception may occur when the actual processing time of a data processing request by a cluster node in the local cluster exceeds a configured request processing time threshold. The actual processing time can be understood as the time it takes for a cluster node in the local cluster to process the currently received data processing request. The configured request processing time threshold can be a pre-set, standard processing time corresponding to the current data processing request. An exception instruction can be used to notify the online service module that an exception has occurred in the processing of the current data processing request by a cluster node in the local cluster.

[0039] The online service module provides online query services, is used to resolve data processing requests associated with the online query services, and creates data processing tasks corresponding to the data processing requests, so that the cluster performs data processing operations based on the data processing tasks. The data processing tasks can be data tasks corresponding to the data processing requests. For example, if the data processing request is a request to query data, the corresponding data processing tasks can be data query tasks.

[0040] The peer cluster and local cluster are typically two big data clusters deployed in different locations. Optionally, the cluster currently connected to the application can be designated as the local cluster, while the cluster not currently connected to the application can be designated as the peer cluster. Generally, the data in the peer and local clusters is identical. Deploying two clusters is for disaster recovery. Specifically, if a data processing request times out or fails in the local cluster's cluster nodes, the request can be handled by the peer cluster's cluster nodes.

[0041] Specifically, the client sends a data processing request to the application program so as to process the data processing request based on the local cluster connected to the application program. In this process, the data processing request can be first parsed and verified by the online service module, so that when the verification result corresponding to the data processing request is normal, the online service module generates a corresponding data processing task based on the data processing request and sends it to the local cluster so that the local cluster processes the data processing task corresponding to the data processing request. When the local cluster processes the data processing task currently received corresponding to the data processing request, the first control module is used to detect whether the cluster node of the local cluster has an abnormality when processing the data processing task corresponding to the data processing request. If a processing abnormality is detected, an abnormal instruction is sent to the online service module so that after receiving the abnormal instruction, the online service module regenerates a data processing task based on the data processing request and sends the regenerated data processing task to the opposite cluster so that the cluster node of the opposite cluster processes the data processing task.

[0042] It should be noted that when the cluster node of the opposite cluster is processing the data processing task, the cluster node of the local cluster is also processing the data processing task.

[0043] Correspondingly, if the first control module does not detect an abnormality in the processing of the data processing request, or detects that the control switch for controlling the cluster node switching to the opposite cluster is in the off state, the data processing request continues to be processed based on the local cluster to obtain a data feedback result corresponding to the data processing request. Among them, the control switch for controlling the cluster node switching to the opposite cluster can be a switch used to control whether the cluster node of the opposite cluster can process the data processing request currently received by the local cluster, that is, a control switch for controlling the cluster direction switching. The control switch can be controlled manually, or it can be opened and closed at regular intervals according to business needs. When the control switch is in the on state, the cluster node of the local cluster and the cluster node of the opposite cluster can process the same data processing request. When the control switch is in the off state, the data processing request is only processed by the cluster node of the local cluster.

[0044] For example, see Figure 2 , Figure 2 This is a structural example diagram of an exception handling method for a big data cluster node. When the online service module receives a data processing request sent by a client, the data processing request is parsed and verified. If the verification result is normal, the online service module sends the data processing task corresponding to the data processing request to the local cluster, so that the cluster node of the local cluster processes the data processing task corresponding to the data processing request. When the cluster node of the local cluster processes the data processing task corresponding to the data processing request, the first control module detects whether the cluster node of the local cluster processes the data processing task abnormally, such as when the cluster node of the local cluster times out in processing the data processing task. If the processing is abnormal, the first control module sends an exception instruction to the online service module, so that the online service module rebuilds the data processing task based on the data processing request, and sends the data processing task to the opposite cluster, so that the data processing task is processed by both the cluster node of the opposite cluster and the cluster node of the local cluster.

[0045] Based on the above, when a data processing request is processed abnormally, while ensuring that the local cluster continues to process the data processing request, the data processing task corresponding to the data processing request is rebuilt so that the opposite cluster can also process the data processing task corresponding to the data processing request, thereby ensuring the stability of data processing. This can be applied to situations where the data processing request is processed abnormally due to a cluster node abnormality of the local cluster, and can also be applied to situations where the local cluster processes the data processing request abnormally due to network jitter. If the data processing request is processed abnormally due to a cluster node abnormality, then when the local cluster cannot feedback the data feedback result corresponding to the data processing request, the opposite cluster can feedback the data feedback result corresponding to the data processing request. If the local cluster processes the data processing request abnormally due to network jitter, when the local cluster has network jitter, the opposite cluster can feedback the corresponding data feedback result; when the opposite cluster has network jitter, the local cluster can feedback the corresponding data feedback result. Based on this, both the normal processing of the data processing request is guaranteed and the processing speed of the data processing request is improved.

[0046] Optionally, the determination based on the first control module that the local cluster has handled the data processing request abnormally is determined in the following manner: based on the first control module, the pre-configured request processing configuration time threshold corresponding to the business type of the request corresponding to the data processing request and / or the cluster node type that processes the data processing request is retrieved from the cache; based on the actual processing time corresponding to the data processing request and the request processing configuration time threshold, it is determined whether the data processing request is handled abnormally.

[0047] Different data processing requests correspond to different business types. Optionally, the business types may be at least two of the following: real-time trading, analytics, streaming, and data query. The configured request processing time threshold may be a pre-set standard time for processing data processing requests based on the processing time requirements of different business types. Different business types may correspond to different configured request processing time thresholds.

[0048] For example, real-time trading services have high requirements for the processing time corresponding to data processing requests, usually in milliseconds. That is, the request processing configuration time threshold for real-time trading services can be set to 50 milliseconds. The processing time requirement for streaming services is usually in the second level, that is, the request processing configuration time threshold for streaming services can be set to 3 seconds. The processing time requirement for data query services is usually in the sub-second level, so the request processing configuration time threshold for data query services can be set to 1 second. The processing time requirement for analytical services is usually in the minute level, so the request processing configuration time threshold for analytical services can be set to 5 minutes.

[0049] Accordingly, the types of cluster nodes that process different data processing requests are different, that is, the cluster node types are different. The cluster node type can be understood as the node type of the cluster nodes contained in the local cluster or the opposite cluster. Optionally, the cluster node types may include: CPU-intensive nodes, GPU / TPU nodes, memory-intensive nodes, storage-intensive nodes, etc. Since cluster nodes of different cluster node types are adapted to data processing requests in different scenarios, the corresponding request processing configuration time threshold can be set according to the scenario adapted by the cluster node type. For example, CPU-intensive nodes can be used to process data processing requests in real-time transaction scenarios, and the request processing configuration time threshold corresponding to the CPU-intensive nodes can be set to 50 milliseconds. Memory-intensive nodes can be used to process data processing requests under streaming processing services, and the request processing configuration time threshold corresponding to the memory-intensive nodes can be set to 3 seconds.

[0050] It should be noted that the request processing configuration time threshold corresponding to the business type of the above different requests and the request processing configuration time threshold corresponding to different cluster node types can be Figure 2 The configuration management module shown in the figure can be used to set the configuration time threshold for request processing, set the on / off state of the control switch used to control the switching to the cluster node of the peer cluster, set the preset number threshold, etc. The set configuration time threshold for request processing can be saved in the cache. The actual processing time can be understood as the actual processing time length of the cluster node of the local cluster for the data processing request.

[0051] Specifically, the first control module determines the business type of the request corresponding to the data processing request currently received by the local cluster and / or the type of cluster node in the local cluster that processes the data processing request. The first control module retrieves from the cache a request processing configuration duration threshold corresponding to the business type and / or cluster node type of the request. When the first control module detects that the actual processing duration of the data processing request by the local cluster exceeds the request processing configuration threshold, it determines that the local cluster has abnormally processed the currently received data processing request.

[0052] For example, see Figure 3 ,pass Figure 2 Before the first control module in the cache retrieves the request processing configuration time threshold corresponding to the business type of the request corresponding to the data processing request and / or the cluster node type processing the data processing request, it can first be Figure 2 The configuration management module in sets the request processing configuration time threshold corresponding to the business type and cluster node type to which each request belongs. Optionally, a global request processing configuration time threshold applicable to all requests can be set, that is, the priority of the global request processing configuration time threshold is the lowest. Set the request processing configuration time threshold applicable to the business type to which each request belongs. It should be noted that the priority of the request processing configuration time threshold applicable to the business type to which each request belongs is higher than the priority of the global request processing configuration time threshold, and the request processing configuration time threshold is applied to the business type to which the specified request belongs.

[0053] Set the request processing configuration time threshold that is suitable for each cluster node type. It should be noted that the priority of the request processing configuration time threshold that is suitable for each cluster node type is higher than the priority of the global request processing configuration time threshold, and the request processing configuration time threshold is applied to the specified cluster node type. In order to facilitate the immediate adjustment of the request processing configuration time threshold, the configuration management module can modify the request processing configuration time threshold and take effect immediately. The configuration management module saves the set request processing configuration time threshold in the cache, so that the first control module retrieves the corresponding request processing configuration time threshold from the cache.

[0054] The first control module reads the request processing configuration time threshold corresponding to the business type or cluster node type of the request corresponding to the current data processing request from the cache, and monitors the actual processing time of the local cluster processing the data processing request. When the actual processing time corresponding to the data processing request exceeds the request processing configuration time threshold, it is determined that there is an abnormality in the cluster node of the local cluster corresponding to the data processing request. Then, while the local cluster continues to process the data processing task corresponding to the data processing request, the first control module can be used to detect whether the control switch for controlling the switching to the cluster node of the opposite cluster is in the open state.

[0055] If the control switch is in the on state, the cluster corresponding to the current data processing request is switched to the peer cluster, and an exception instruction is sent to the online service module to enable the online service module to rebuild the data processing task corresponding to the data processing request and send it to the peer cluster. Figure 3 Based on this, both the local cluster and the peer cluster can process the data processing request. Since the switching granularity is request, only the cluster pointing to the current data processing request is changed, and the overall cluster pointing is not changed. It is stateless and does not require backswitching. That is, other data processing requests are still processed by the local cluster. It should be noted that the speed of switching to the peer cluster depends on the pre-configured request processing configuration time threshold. That is, the smaller the request processing configuration time threshold, the faster the cluster pointing switch.

[0056] It should be noted that whether the control switch for controlling the cluster node switching to the peer cluster is in the on state can be pre-set by the configuration management module. Optionally, the control switch for automatic cluster switching can be set to be in the on state according to actual conditions, or the control switch can be set to be in the on state at a scheduled time.

[0057] Correspondingly, if the actual processing time does not exceed the request processing configuration time threshold, or the control switch is in the off state, the data processing task corresponding to the data processing request is blocked and waits for processing by the cluster node of the local cluster.

[0058] By judging whether the actual processing time exceeds the configured request processing time threshold, it is determined whether the cluster node of the local cluster is abnormal, so that the cluster direction is switched for the data processing request that exceeds the configured request processing time threshold, reducing the impact range of the cluster switching and ensuring the normal processing of other data processing requests.

[0059] S120: Receive a data processing task based on the peer cluster, and process the data processing task based on the cluster node of the peer cluster.

[0060] Specifically, when the peer cluster receives the data processing task corresponding to the data processing request sent by the online service module, it determines the cluster node of the peer cluster corresponding to the data processing task and processes the data processing task through the cluster node.

[0061] S130: When the first control module detects that the local cluster and / or the opposite cluster have output data feedback results corresponding to the data processing request, send a task stop instruction to the cluster that has not output the data feedback result.

[0062] The data feedback result can be understood as the processing result corresponding to the data processing request. For example, if the data processing request is a request to query business data, the data feedback result can be the corresponding business data. The task stop instruction can be used to notify the cluster that has not output the data feedback result to terminate processing of the data processing request.

[0063] Specifically, when the first control module detects that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request, in order to avoid wasting the cluster's computing resources, a task stop instruction can be sent to the cluster that has not output the data feedback result, so that the cluster that has not output the data feedback result stops processing the data processing task in time, making it convenient for the cluster that has not output the data feedback result to process other data processing requests, thereby improving the computing resource utilization of the cluster.

[0064] Optionally, when a processing exception of a data processing request is detected, processing process data associated with the data processing request is recorded.

[0065] The processing data may include processing data when the local cluster processes a data processing request and / or processing data when the peer cluster processes a data processing request. Optionally, the processing data may include: processing data when the local cluster processes the data processing request before switching to the peer cluster; and processing data when the local cluster and the peer cluster process the data processing request after switching to the peer cluster.

[0066] The processing data that has not been switched to the peer cluster includes at least one of: request processing data corresponding to the local cluster's processing of the data processing request and data feedback results fed back by the local cluster; the processing data that has been switched to the peer cluster includes at least one of: data feedback results fed back by the local cluster, data feedback results fed back by the peer cluster, request processing data corresponding to the peer cluster's processing of the data processing request, and request processing data corresponding to the local cluster's processing of the data processing request. Request processing data can be understood as data information corresponding to the processing of a data processing request.

[0067] Specifically, when a data processing request is processed abnormally, there are multiple situations. One situation is: the actual processing time of the local cluster processing the data processing request exceeds the configured request processing time threshold, and the control switch of the cluster node used to control the switch to the peer cluster is in the open state, and the cluster corresponding to the data processing request is switched to the peer cluster. In this case, if the data feedback result is provided by the local cluster, the processing process data may include: the data feedback result provided by the local cluster, the request processing data when the peer cluster processes the data processing request, and the request processing data corresponding to the local cluster processing the data processing request. Optionally, an automatic switching exception log indicating successful processing by the local cluster can be generated based on the data feedback result provided by the local cluster, the request processing data when the peer cluster processes the data processing request, and the request processing data corresponding to the local cluster processing the data processing request. In this case, if the data feedback result is provided by the peer cluster, the processing process data may include: the data feedback result provided by the peer cluster, the request processing data when the local cluster processes the data processing request, and the request processing data corresponding to the peer cluster processing the data processing request. Optionally, an automatic switching exception log indicating successful processing by the peer cluster can be generated based on the data feedback results fed back by the peer cluster, the request processing data when the local cluster processes the data processing request, and the request processing data corresponding to the peer cluster's processing of the data processing request. In this case, if the peer cluster also times out when processing the data processing request, the processing process data may include: the data feedback results fed back by the local cluster and / or the peer cluster, the request processing data corresponding to the timeout when the local cluster processes the data processing request, and the request processing data corresponding to the timeout when the peer cluster processes the data processing request. Accordingly, an automatic switching exception log indicating a dual-end cluster processing timeout can be generated based on the aforementioned processing process data.

[0068] Another situation in which a data processing request is handled abnormally is that the actual processing time of the data processing request handled by the local cluster exceeds the request processing configuration time threshold, but the control switch of the cluster node used to control the switch to the opposite cluster is in the off state, and the cluster corresponding to the data processing request points to the opposite cluster. In this case, if the local cluster feeds back the data feedback result, the processing process data may include: the request processing data corresponding to the data processing request processed by the local cluster and the data feedback result fed back by the local cluster. Optionally, an exception log of the local cluster processing timeout can be generated based on the request processing data corresponding to the data processing request processed by the local cluster and the data feedback result fed back by the local cluster. If the local cluster does not feed back the data feedback result, the processing process data may include: the request processing data corresponding to the data processing request processed by the local cluster. Optionally, an exception log of the failure of the local cluster processing can be generated based on the request processing data corresponding to the data processing request processed by the local cluster. The exception log corresponding to the above-mentioned processing process data is recorded and processed.

[0069] For example, see Figure 4 , Figure 4 This is an example diagram of the process of recording exception logs. When a data processing request is detected to be abnormal or the data feedback result corresponding to the data processing request is abnormal, for example, the data processing request is timed out, the data feedback result corresponding to the data processing request is parsed incorrectly, or the data processing result feedback of the local cluster is limited, etc., you can use Figure 2 The exception log processing module in the generates exception logs based on the processing data corresponding to the request exception processing, so as to facilitate subsequent exception analysis. If it is detected that the cluster corresponding to the data processing request has switched, it can be Figure 2 The abnormal log processing module in the automatic switching abnormal log records. Optionally, the log types of the automatic switching abnormal log may include: local cluster processing success, peer cluster processing success, dual-end processing timeout, peer cluster processing abnormality, etc.

[0070] See also Figure 4 After the local cluster and / or the opposite cluster feedback the data feedback result, determine whether the cluster direction switching occurs during the processing of this data processing request. If it is determined that the cluster direction is switched from the local cluster to the opposite cluster, determine the business type and / or cluster node type of the request corresponding to the data processing request to call the corresponding automatic switching exception log template. Through the automatic switching exception log template and the processing process data corresponding to the data processing request, the corresponding automatic switching exception log is generated and recorded. Based on this, it is convenient to Figure 2The anomaly detection module in the function performs statistical analysis on the recorded automatic switching anomaly logs. If it is determined that the cluster pointer has not switched, but an anomaly occurs during the processing of a data processing request, a general anomaly log is generated and recorded based on the processing data corresponding to the data processing request.

[0071] The technical solution of this embodiment is to send an exception instruction to the online service module when the first control module detects that the data processing request currently received by the local cluster is processed abnormally, so that the online service module rebuilds the data processing task based on the data processing request and sends the data processing task to the opposite cluster, so as to facilitate the normal processing of the data processing task corresponding to the data processing request by the cluster node of the opposite cluster, thereby ensuring the processing time of the data processing request and avoiding the problem of the data processing request processing time being too long or the data processing request being unable to be responded to. When the opposite cluster receives the data processing task, the data processing task is processed based on the cluster node of the opposite cluster. That is, while the cluster node of the local cluster processes the data processing request, the cluster node of the opposite cluster processes the data processing request, thereby ensuring the processing speed of the data processing request. When the first control module detects that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result, so as to avoid wasting the computing resources of the cluster that has not output the data feedback result, facilitate the cluster that has not output the data feedback result to process other data processing requests, and thus improve the computing resource utilization of the cluster. The present invention solves the problem of increased processing time of all requests caused by overall cluster switching in the prior art, shortens the processing time of data processing requests when the cluster node of the local cluster is abnormal, and avoids the problem of data processing requests being unable to respond due to abnormal cluster nodes. When the cluster node of the local cluster processes the data processing request abnormally, the data processing request is processed by both the cluster node of the opposite cluster and the cluster node of the local cluster, thereby improving the processing speed of the data processing request. Moreover, the above process only affects the data processing request accessing the current cluster node of the local cluster, and does not affect the data processing request accessing other cluster nodes of the local cluster, reducing the impact range of the abnormal cluster node and ensuring the processing speed of data processing requests accessing other cluster nodes of the local cluster.

[0072] Example 2

[0073] Figure 5 This is a flowchart of a method for handling exceptions in a big data cluster node provided by the second embodiment of the present invention. This embodiment of the present invention is a preferred embodiment of the above-mentioned invention embodiment. For its specific implementation, please refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here. Figure 5 As shown, the method includes:

[0074] S210: When the online service module receives a data processing request, the data processing request is parsed to obtain the request type, the business type, and field information of the data processing request.

[0075] The request type can be understood as the type of data processing request, for example, an HTML request, an API request, or a file upload request. The business type of the request can be understood as the business type corresponding to the data processing request. Optionally, the business type of the request can be a real-time transaction business, an analytics business, a streaming business, a data query business, or other business types. The field information can be understood as the request content and other configuration field information corresponding to the data processing request.

[0076] Specifically, when a data processing request is received through the online service module, the data processing request can be parsed to determine the request type, business type and field information corresponding to the data processing request, so as to verify the request type, business type and field information, determine the compliance of the data processing request and ensure the security of the cluster.

[0077] S220: Verify the parsed field information corresponding to the data processing request according to a pre-configured verification field corresponding to the request type and / or the business type to which the request belongs, and obtain a verification result.

[0078] The check field may be a pre-configured field used to check field information of the data processing request. The check result may be a result indicating that the data processing request has passed the check or a result indicating that the data processing request has failed the check.

[0079] Specifically, a check field that matches the request type and / or business type of the data processing request is determined from pre-configured check fields corresponding to the request type and the business type of the request. Field information in the data processing request is verified using the check field. If the check field matches the field information in the data processing request, the verification result is determined to be a passed data processing request. Conversely, if the check field does not match the field information in the data processing request, the verification result is determined to be a failed data processing request.

[0080] S230: When it is detected that the verification result meets the preset condition, the data processing request is sent to the local cluster, so that the data processing request is processed based on the cluster nodes in the local cluster.

[0081] The preset condition may be that the check field matches the field information of the data processing request.

[0082] Specifically, if the verification field matches the field information in the data processing request, the verification result is determined to be a passed verification of the data processing request, i.e., the verification result satisfies the preset conditions. If the verification result satisfies the preset conditions, the data processing request is sent to the local cluster, so that the cluster node of the local cluster processes the data processing request.

[0083] For example, see Figure 6 , Figure 6 This is an example diagram of a process for verifying a data processing request. When the online service module receives a data processing request, it reads the pre-configured configuration information for parsing the data processing request, and parses and processes the data processing request based on the configuration information to determine the request type, the business type, and the field information corresponding to the data processing request. Determine the verification rules corresponding to the request type and / or the business type of the data processing request, that is, the verification field mentioned in the embodiment of the present invention. Verify the field information of the data processing request using the verification rules to determine whether the verification result meets the preset conditions. If the verification result meets the preset conditions, the verification passes, and the data processing task corresponding to the data processing request is sent to the local cluster, so that the local cluster processes the data processing request. Correspondingly, if the verification result does not meet the preset conditions, the verification fails, and the verification failure result is returned, and the processing ends.

[0084] S240. When the first control module detects an exception in processing a data processing request currently received by the local cluster, send an exception instruction to the online service module so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster.

[0085] S250: Receive a data processing task based on the peer cluster, and process the data processing task based on the cluster node of the peer cluster.

[0086] S260: When the first control module detects that the local cluster and / or the opposite cluster has output a data feedback result corresponding to the data processing request, send a task stop instruction to the cluster that has not output the data feedback result.

[0087] Optionally, the method also includes: based on the anomaly detection module, when it detects that the number of request anomaly processing of the business type and / or cluster node type of any data processing request reaches a preset threshold, recording the business type and / or cluster node type to which the request belongs; and, based on the business type and / or cluster node type to which the request belongs, generating early warning information and sending it to the target terminal device.

[0088] The anomaly detection module is used to detect whether the number of request anomalies exceeds a preset threshold. The number of request anomaly handling can be understood as the number of detected request anomaly handling within a target duration. Alternatively, when feedback processing based on an early warning message is detected, the number of request anomaly handling that generated the early warning message is reset to zero and the number of request anomaly handling is re-accumulated. The target duration can be a pre-set duration for counting the number of request anomaly handling. The preset threshold can be a pre-set standard value for the number of request anomaly handling.

[0089] Warning information can be used to warn of the business type and / or cluster node type of requests whose number of abnormal request handling exceeds a preset threshold. The target terminal device can be the terminal device of the corresponding staff member or the corresponding monitoring platform. Optionally, the staff member's terminal device can be an electronic device such as a mobile phone or computer. Optionally, the warning information can be sent to the terminal device of the corresponding staff member via email, phone call, text message, etc.

[0090] Specifically, when the first control module detects that the number of abnormal processing requests for any data processing request of a business type and / or cluster node type reaches a preset threshold within a target duration or after the last warning information feedback, the business type and / or cluster node type of the request is recorded. A warning prompt message is generated based on the business type and / or cluster node type of the request and sent to the target terminal device.

[0091] In an embodiment of the present invention, when the number of request exception processing reaches a preset number threshold, the method of recording the business type and / or cluster node type to which the request belongs can be: when the statistical condition of the number of request exception processing is detected to be met based on the exception detection module, obtaining associated information of at least one request exception processing associated with the statistical condition; wherein the request exception processing corresponds to the processing exception of the local cluster on the data processing request, and / or the local cluster or the opposite cluster does not feedback the data feedback result corresponding to the data processing request; for at least one request exception processing, according to the business type and / or cluster node type of the request in the associated information corresponding to the request exception processing, the number of request exception processing corresponding to the business type and / or cluster node type of the request is accumulated; when the accumulated number of request exception processing corresponding to the business type and / or cluster node type of the request reaches the corresponding preset number threshold, the business type and / or cluster node type of the request is recorded; when the accumulated number of request exception processing corresponding to the business type and / or cluster node type of the request does not reach the corresponding preset number threshold, the next request exception processing is continued.

[0092] Request exception handling can be understood as a data processing request being handled abnormally. This occurs when the local cluster handles the data processing request abnormally and / or when the local cluster or the peer cluster fails to provide the data feedback corresponding to the data processing request. The statistical condition for satisfying the number of request exception handling counts can be a preset time interval between the current moment and the last time a request exception was detected, or the number of request exception handling counts corresponding to the last warning message has been reset to zero.

[0093] The associated information of at least one request exception handling may include the business type of the data processing request and / or the type of cluster node that processes the data processing request. Optionally, the associated information corresponding to the request exception handling may be obtained from a general exception log or an automatic switching exception log.

[0094] Specifically, when the anomaly detection module detects that the current time is separated from the time corresponding to the last request anomaly handling detection by a preset time length, or when the number of request anomaly handling corresponding to the previous warning information has been reset to zero, it is determined that the statistical condition for the number of request anomaly handling is satisfied, and then at least one request anomaly handling-related information corresponding to the statistical condition is obtained. Based on the business type and / or cluster node type of the request in the request anomaly handling-related information, the number of request anomaly handling corresponding to the business type and / or cluster node type of the same request is accumulated. When the number of request anomaly handling corresponding to the business type and / or cluster node type obtained by accumulation within the target time length reaches the corresponding preset number threshold, or when the number of request anomaly handling corresponding to the business type and / or cluster node type obtained by accumulation reaches the corresponding preset number threshold, the business type and / or cluster node type of the request is recorded, and an warning information is generated based on the business type and / or cluster node type of the request for warning processing. If the number of request anomaly handling corresponding to the business type and / or cluster node type obtained by accumulation within the target time length reaches the corresponding preset number threshold, the next request anomaly handling is processed.

[0095] For example, see Figure 7, request exception processing can be that the data processing request is processed abnormally when a cluster switch occurs. The anomaly detection module can perform a count of the number of request exception processing every 5 minutes. Obtain the associated information of the request exception processing within five minutes, and count the business types of the requests corresponding to the request exception processing in which the cluster pointing switch occurs. Determine the number of request exception processing times for the business type to which the first request belongs within five minutes. When the number of request exception processing times exceeds the preset number threshold, the business type to which the request belongs is added to the warning list. If there is a business type to which the next request belongs, determine the number of request exception processing times for the business type to which the next request belongs within five minutes, and when the number of request exception processing times exceeds the preset number threshold, add the business type to which the request belongs to the warning list until the number of request exception processing times corresponding to all business types to which requests belong within five minutes is completed. Based on the business types to which all requests belong in the warning list within five minutes, generate warning information and perform warning processing.

[0096] The technical solution of this embodiment is to parse the data processing request when the online service module receives it to obtain the request type, business type, and field information of the data processing request, and verify the parsed field information corresponding to the data processing request based on the pre-configured verification field corresponding to the request type and / or business type to obtain a verification result. When it is detected that the verification result meets the preset conditions, the data processing request is sent to the local cluster so that the data processing request is processed by the cluster nodes in the local cluster, ensuring the compliance of the data processing request and the security of the cluster. When the first control module detects that the processing of the data processing request currently received by the local cluster is abnormal, it sends an exception instruction to the online service module so that the online service module reconstructs the data processing task based on the data processing request and sends the data processing task to the peer cluster, facilitating the normal processing of the data processing task corresponding to the data processing request by the cluster nodes of the peer cluster, ensuring the processing time of the data processing request, and avoiding the problem of excessive processing time or inability to respond to the data processing request. When the peer cluster receives the data processing task, the cluster nodes of the peer cluster process the data processing task. That is, while the cluster nodes of the local cluster process the data processing request, the cluster nodes of the opposite cluster process the data processing request, thereby ensuring the processing speed of the data processing request. When the first control module detects that the local cluster and / or the opposite cluster have output the data feedback result corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result, so as to avoid the waste of computing resources of the cluster that has not output the data feedback result, and facilitate the cluster that has not output the data feedback result to process other data processing requests, thereby improving the computing resource utilization of the cluster. The present invention solves the problem of the increase in the processing time of all requests caused by the overall switching of clusters in the prior art, shortens the processing time of data processing requests when the cluster nodes of the local cluster are abnormal, and avoids the problem of the inability to respond to data processing requests caused by abnormal cluster nodes. When the cluster nodes of the local cluster process the data processing request abnormally, the data processing request is processed by both the cluster nodes of the opposite cluster and the cluster nodes of the local cluster, thereby improving the processing speed of the data processing request. Moreover, the above process only affects the data processing requests of the current cluster node accessing the local cluster, and does not affect the data processing requests of other cluster nodes accessing the local cluster, thereby reducing the impact range of the abnormal cluster node and ensuring the processing speed of data processing requests of other cluster nodes accessing the local cluster.

[0097] Example 3

[0098] Figure 8This is a flowchart of an exception handling method for a big data cluster node provided by the third embodiment of the present invention. The embodiment of the present invention is an example of the above embodiment. For its specific implementation, please refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 8 As shown, the method includes:

[0099] S310: When receiving an online query request, the online service module parses and verifies the online query request to obtain a verification result.

[0100] The online query request corresponds to the data processing request mentioned in the above embodiment.

[0101] Specifically, when the online service module receives an online query request from a client, it parses the request and determines the corresponding request parameters. The request parameters are then verified using validation rules to obtain a validation result. It should be noted that the validation rules correspond to the validation fields mentioned in the above embodiment. The request parameters correspond to the request type, service type, and field information of the data processing request mentioned in the above embodiment.

[0102] S320. When the verification result meets the preset conditions, the online service module obtains the current cluster pointing, creates a data query task corresponding to the online query request, and sends the data query task to the local cluster corresponding to the current cluster pointing, so as to process the data query task based on the cluster nodes in the local cluster.

[0103] The fact that the verification result meets the preset conditions can be interpreted as the online query request passing verification. The current cluster can refer to the cluster currently associated with the application. In other words, it refers to the cluster currently processing all online query requests. The data query task corresponds to the data processing task mentioned in the above embodiment.

[0104] Specifically, if the verification result indicates that the online query request has passed verification, the online service module determines that the verification result meets the preset conditions, obtains the current cluster reference, creates a data query task corresponding to the online query request, and sends the data query task to the local cluster corresponding to the current cluster reference, so that the data query task is processed by the cluster nodes of the local cluster.

[0105] S330. Detect, through the first control module, whether the data query task processed by the local cluster is abnormal. When it is detected that the data query task processed by the local cluster is abnormal and the automatic switching switch is in the on state, rebuild the data query task and send the data query task to the opposite cluster so that the data query task is processed based on the cluster node of the opposite cluster.

[0106] The automatic switching switch corresponds to the control switch mentioned in the above embodiment.

[0107] Specifically, in the process of processing data query tasks based on the local cluster, the first control module is used to detect whether the local cluster processes the data query task abnormally. When it is detected that the local cluster processes the data query task abnormally and the automatic switching switch is in the on state, it is determined that the switching conditions for the cluster pointing are currently met, and the cluster pointing corresponding to the current online query request is switched to the opposite cluster, and the data query task corresponding to the current online query request is sent to the opposite cluster to process the data query task through the cluster node of the opposite cluster.

[0108] S340: When the first control module detects that the local cluster and / or the opposite cluster have returned query results corresponding to the data query task, it sends a task stop instruction to the cluster that has not output the query result.

[0109] The query result corresponds to the data feedback result mentioned in the above embodiment.

[0110] Specifically, when the first control module detects that the local cluster and / or the opposite cluster have returned the query results corresponding to the data query task within the preset waiting time, a task stop instruction is sent to the cluster that has not output the query result. And detect whether the query result is fed back by the opposite cluster. If the query result is fed back by the opposite cluster, the corresponding automatic switching exception log is recorded. If the query result is fed back by the local cluster, the corresponding automatic switching exception log is recorded. If the first control module does not detect that the local cluster and the opposite cluster have returned the query results corresponding to the data query task within the preset waiting time, it is determined that the query of the local cluster and the opposite cluster has timed out, and the corresponding automatic switching exception log is recorded.

[0111] In an embodiment of the present invention, when an online query request is received by an online service module, the online query request is parsed and verified to obtain a verification result. When the verification result meets the preset conditions, the online service module obtains the current cluster pointer, creates a data query task corresponding to the online query request, and sends the data query task to the local cluster corresponding to the current cluster pointer, so that the data query task is processed based on the cluster nodes in the local cluster. The compliance of the online query request and the security of the cluster are guaranteed. The first control module detects whether the local cluster processes the data query task abnormally. When it is detected that the local cluster processes the data query task abnormally and the automatic switching switch is in the on state, the data query task is rebuilt and sent to the opposite cluster, so that the data query task is processed based on the cluster nodes of the opposite cluster, facilitating the normal processing of the data query task by the cluster nodes of the opposite cluster, ensuring the task processing time, and avoiding the problem of excessive task processing time or inability to process in a timely manner. When the first control module detects that the local cluster and / or the opposite cluster have returned the query result corresponding to the data query task, a task stop instruction is sent to the cluster that has not output the query result. This avoids wasting computing resources in clusters that haven't output query results, facilitating the processing of other online query requests by clusters that haven't output query results, thereby improving cluster computing resource utilization. The present invention addresses the prior art issue of increased processing time for all requests caused by switching clusters overall. It shortens the processing time for online query requests when a cluster node in the local cluster is abnormal, avoids the problem of online query requests being unresponsive due to abnormal cluster nodes, reduces the impact of abnormal cluster nodes, and ensures the processing speed of online query requests accessing other cluster nodes in the local cluster.

[0112] Example 4

[0113] Figure 9 This is a schematic diagram of the structure of an exception handling device for a big data cluster node provided by the fourth embodiment of the present invention. Figure 9 As shown, the device includes: an abnormal instruction sending module 410, a task processing module 420 and a task stopping module 430.

[0114] The exception instruction sending module 410 is used to send an exception instruction to the online service module when the first control module detects that the data processing request currently received by the local cluster is processed abnormally, so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster; the task processing module 420 is used to receive the data processing task based on the opposite cluster and process the data processing task based on the cluster node of the opposite cluster; the task stopping module 430 is used to send a task stopping instruction to the cluster that has not output the data feedback result when the first control module detects that the local cluster and / or the opposite cluster have output the data feedback result corresponding to the data processing request.

[0115] The technical solution of this embodiment is to send an exception instruction to the online service module when the first control module detects that the data processing request currently received by the local cluster is processed abnormally, so that the online service module rebuilds the data processing task based on the data processing request and sends the data processing task to the opposite cluster, so as to facilitate the normal processing of the data processing task corresponding to the data processing request by the cluster node of the opposite cluster, thereby ensuring the processing time of the data processing request and avoiding the problem of the data processing request processing time being too long or the data processing request being unable to be responded to. When the opposite cluster receives the data processing task, the data processing task is processed based on the cluster node of the opposite cluster. That is, while the cluster node of the local cluster processes the data processing request, the cluster node of the opposite cluster processes the data processing request, thereby ensuring the processing speed of the data processing request. When the first control module detects that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result, so as to avoid wasting the computing resources of the cluster that has not output the data feedback result, facilitate the cluster that has not output the data feedback result to process other data processing requests, and thus improve the computing resource utilization of the cluster. The present invention solves the problem of increased processing time of all requests caused by overall cluster switching in the prior art, shortens the processing time of data processing requests when the cluster node of the local cluster is abnormal, and avoids the problem of data processing requests being unable to respond due to abnormal cluster nodes. When the cluster node of the local cluster processes the data processing request abnormally, the data processing request is processed by both the cluster node of the opposite cluster and the cluster node of the local cluster, thereby improving the processing speed of the data processing request. Moreover, the above process only affects the data processing request accessing the current cluster node of the local cluster, and does not affect the data processing request accessing other cluster nodes of the local cluster, reducing the impact range of the abnormal cluster node and ensuring the processing speed of data processing requests accessing other cluster nodes of the local cluster.

[0116] Based on the above embodiment, optionally, the exception instruction sending module includes a request processing exception determination unit, which is used to retrieve from the cache, based on the first control module, a pre-configured request processing configuration time threshold corresponding to the business type of the request corresponding to the data processing request and / or the cluster node type that processes the data processing request; and determine whether the data processing request is processed abnormally based on the actual processing time corresponding to the data processing request and the request processing configuration time threshold.

[0117] Optionally, the device also includes: an abnormal processing number warning module, which includes: a type recording unit, which is used to record the business type and / or cluster node type of the request when the abnormal detection module detects that the number of abnormal processing requests of the business type and / or cluster node type of any data processing request reaches a preset number threshold; and an early warning information sending unit, which is used to generate early warning information based on the business type and / or cluster node type of the request and send it to the target terminal device.

[0118] Optionally, a type recording unit is used to obtain associated information of at least one request exception processing associated with the statistical condition when the statistical condition of the number of request exception processing is detected to be met based on the exception detection module; wherein the request exception processing corresponds to the processing exception of the data processing request by the local cluster, and / or the local cluster or the opposite cluster does not feedback the data feedback result corresponding to the data processing request; for at least one request exception processing, the number of request exception processings corresponding to the business type and / or cluster node type of the request in the associated information corresponding to the request exception processing is accumulated; when the accumulated number of request exception processings corresponding to the business type and / or cluster node type of the request reaches the corresponding preset number threshold, the business type and / or cluster node type of the request is recorded; when the accumulated number of request exception processings corresponding to the business type and / or cluster node type of the request does not reach the corresponding preset number threshold, the next request exception processing is continued.

[0119] Optionally, the device further includes: a processing process data recording module, configured to record processing process data associated with the data processing request when a processing exception of the data processing request is detected.

[0120] Optionally, a processing process data recording module is used to record the processing process data of data processing requests based on the local cluster when not switching to the opposite cluster; and to record the processing process data of data processing requests based on the local cluster and the opposite cluster when switching to the opposite cluster.

[0121] Optionally, the processing process data that is not switched to the opposite cluster includes: request processing data corresponding to the data processing request processed by the local cluster, and at least one of the data feedback results fed back by the local cluster; the processing process data that is switched to the opposite cluster includes: data feedback results fed back by the local cluster, data feedback results fed back by the opposite cluster, request processing data corresponding to the data processing request processed by the opposite cluster, and at least one of the request processing data corresponding to the data processing request processed by the local cluster.

[0122] Optionally, the device also includes: a data processing request verification module, which is used to parse the data processing request when the data processing request is received based on the online service module to obtain the request type, the business type of the request and the field information of the data processing request; verify the parsed field information corresponding to the data processing request according to the pre-configured verification field of the request type and / or the business type of the request to obtain a verification result; when it is detected that the verification result meets the preset conditions, the data processing request is sent to the local cluster to process the data processing request based on the cluster node in the local cluster.

[0123] The exception handling device for a big data cluster node provided by an embodiment of the present invention can execute the exception handling method for a big data cluster node provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0124] Example 5

[0125] Figure 10 1 is a structural diagram of an electronic device provided in Example 5 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0126] like Figure 10 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0127] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0128] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the exception handling method for a big data cluster node.

[0129] In some embodiments, the exception handling method for a big data cluster node may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the exception handling method for a big data cluster node described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the exception handling method for a big data cluster node by any other appropriate means (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the exception handling method for a big data cluster node of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0133] Example 6

[0134] Embodiment 6 of the present invention further provides a computer-readable storage medium storing computer instructions, the computer instructions being used to cause a processor to execute a method for handling exceptions in a big data cluster node, the method comprising:

[0135] When the first control module detects that the processing of the data processing request currently received by the local cluster is abnormal, an abnormal instruction is sent to the online service module to enable the online service module to rebuild the data processing task based on the data processing request and send it to the opposite cluster; the data processing task is received based on the opposite cluster, and the data processing task is processed based on the cluster node of the opposite cluster; when the first control module detects that the local cluster and / or the opposite cluster have output data feedback results corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result.

[0136] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0140] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0141] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for handling exceptions in a big data cluster node, characterized in that: The method comprises: When the first control module detects that a data processing request currently received by the local cluster is processed abnormally, the control module sends an abnormal instruction to the online service module, so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster; receiving the data processing task based on the peer cluster, and processing the data processing task based on a cluster node of the peer cluster; When it is detected based on the first control module that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request, a task stop instruction is sent to the cluster that has not output the data feedback result.

2. The method according to claim 1, characterized in that Determining, based on the first control module, that the local cluster handles the data processing request abnormally is determined in the following manner: Retrieving from the cache, based on the first control module, a pre-configured request processing configuration time threshold corresponding to the business type of the request corresponding to the data processing request and / or the type of cluster node processing the data processing request; Determine whether the data processing request is processed abnormally according to the actual processing time corresponding to the data processing request and the request processing configuration time threshold.

3. The method according to claim 1, characterized in that The method further comprises: When the anomaly detection module detects that the number of abnormal processing requests for the business type and / or cluster node type of any data processing request reaches a preset number threshold, the anomaly detection module records the business type and / or cluster node type of the request; and Based on the business type and / or cluster node type to which the request belongs, an early warning message is generated and sent to the target terminal device.

4. The method according to claim 3, characterized in that The first control module, when detecting that the number of times of abnormal processing of a business type and / or cluster node type of any data processing request reaches a preset number threshold, records the business type and / or cluster node type of the request, including: When a statistical condition of the number of request exception processing is detected to be satisfied based on the anomaly detection module, associated information of at least one request exception processing associated with the statistical condition is obtained; wherein the request exception processing corresponds to an abnormal processing of the data processing request by the local cluster, and / or the local cluster or the peer cluster fails to feedback a data feedback result corresponding to the data processing request; For the at least one request exception processing, according to the business type and / or cluster node type of the request in the associated information corresponding to the request exception processing, the number of request exception processing times corresponding to the business type and / or cluster node type of the request is accumulated; When the accumulated number of request exception handling times corresponding to the business type and / or cluster node type of the request reaches the corresponding preset number threshold, the business type and / or cluster node type of the request is recorded; When the accumulated number of request exception processing times corresponding to the business type and / or cluster node type of the request does not reach the corresponding preset number threshold, the next request exception processing is continued.

5. The method according to claim 1, wherein The method further comprises: When a data processing request processing exception is detected, processing process data associated with the data processing request is recorded.

6. The method according to claim 5, characterized in that The recording of processing data associated with the data processing request includes: Processing data of the data processing request processed by the local cluster when the local cluster has not switched to the peer cluster; When switching to the opposite cluster, processing process data of the data processing request when the local cluster and the opposite cluster are processed is used.

7. The method according to claim 5 or 6, characterized in that The processing data not switched to the peer cluster includes at least one of request processing data corresponding to the data processing request processed by the local cluster and a data feedback result fed back by the local cluster; The processing data of switching to the opposite cluster includes: the data feedback result fed back by the local cluster, the data feedback result fed back by the opposite cluster, the request processing data corresponding to the data processing request processed by the opposite cluster, and at least one of the request processing data corresponding to the data processing request processed by the local cluster.

8. The method according to claim 1, characterized in that The method further comprises: When the data processing request is received based on the online service module, the data processing request is parsed to obtain the request type, the business type and field information of the data processing request; Verifying the parsed field information corresponding to the data processing request according to a pre-configured verification field corresponding to the request type and / or the business type to which the request belongs, to obtain a verification result; When it is detected that the verification result meets a preset condition, the data processing request is sent to the local cluster, so that the data processing request is processed based on the cluster nodes in the local cluster.

9. An exception handling device for a big data cluster node, characterized in that: include: an abnormal instruction sending module, configured to send an abnormal instruction to the online service module when the first control module detects an abnormality in processing a data processing request currently received by the local cluster, so that the online service module rebuilds the data processing task based on the data processing request and sends it to the opposite cluster; a task processing module, configured to receive the data processing task based on the peer cluster, and process the data processing task based on the cluster node of the peer cluster; The task stopping module is configured to send a task stopping instruction to the cluster that has not output the data feedback result when the first control module detects that the local cluster and / or the opposite cluster has output the data feedback result corresponding to the data processing request.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the exception handling method for big data cluster nodes according to any one of claims 1 to 8.

Images