Data processing method and device, equipment and medium

Abstract

The invention relates to the field of data processing, in particular to a data processing method and device, equipment and a medium, which are used for improving the efficiency of searching a target node and limiting the search depth so as to ensure the search security. The method comprises the following steps: detecting an alarm node in the batch processing system, taking the alarm node as a search starting node, and repeatedly executing the following operations: searching a front key node of the search starting node by taking the search starting node as a starting point in a node network established based on the successive relationship of a plurality of processing nodes in the process of executing business processing, determining a target node; if the number of the searched front key nodes reaches a set threshold value, the current search process is ended, and a search result is obtained; if the search result comprises the to-be-searched node, taking the to-be-searched node as a new search starting node until the search result does not comprise the to-be-searched node; and obtaining a target search result corresponding to the alarm node based on the target node obtained in each search process.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a data processing method, device, equipment and medium. Background Art

[0002] With the continuous development of computer technology, the use of computers for batch business processing has greatly improved the speed of business processing, saved manpower related to business processing, and has become the first choice for batch business processing in all walks of life.

[0003] As a post-data processing method, batch business processing has the characteristics of large concurrency, long time consumption and complicated processing logic. If the data volume increases, batch business errors and upstream system data supply delays occur, batch business delays (referred to as batch delays) may occur. In order to avoid batch delays affecting the availability of application online business, it is necessary to locate the cause of the delay in a timely manner, solve the problem and resume batch operation.

[0004] Currently, in most cases, the batch businesses being run are complex and intertwined. If an upstream task exception causes a delay in a downstream task, it is usually necessary to manually check each level to find the root cause, which is inefficient. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, apparatus, device and medium for improving the efficiency of searching for target nodes and limiting the search depth to ensure search security.

[0006] In a first aspect, the present application provides a data processing method, the method comprising:

[0007] Detecting an alarm node in a batch processing system, wherein the batch processing system includes a plurality of processing nodes, and the alarm node is a processing node that triggers an alarm condition among the plurality of processing nodes;

[0008] Take the alarm node as the search start node and repeat the following operations:

[0009] In a node network established based on the sequential relationship of the multiple processing nodes in the process of executing business processing, starting from the search start node, searching for the previous key node of the search start node to determine a target node, the target node being the previous key node that causes the search start node to trigger an alarm condition;

[0010] If the number of searched front key nodes reaches the set threshold, the current search process ends and the search result is obtained;

[0011] If the search result includes a node to be searched, the node to be searched is used as a new search start node until the search result does not include the node to be searched, wherein the node to be searched is a previous key node that has not been searched in the target layer and a previous key node that is in a waiting state, and the target layer is the layer where the last searched previous key node is located or the next layer of the layer;

[0012] Based on the target node obtained in each search process, a target search result corresponding to the alarm node is obtained.

[0013] In an embodiment of the present application, after detecting the alarm node in the batch processing system, the alarm node is used as the search start node, and the following operations are repeatedly performed: in a node network established based on the sequential relationship of multiple processing nodes in the execution of business processing, the search start node is used as the starting point, and the front key node of the search start node is searched to determine the target node that causes the search start node to trigger the alarm condition; if the number of the searched front key nodes reaches the set threshold, the current search process is terminated, and the search results are obtained to limit the search depth, reduce the occupied memory resources, reduce the occurrence of problems such as memory overflow and stack overflow, improve the search security, and improve the efficiency of searching the target node; if the search results include the node to be searched, it means that the node network has not yet completed the complete retrieval, and some nodes need to be retrieved. Therefore, in order to ensure the comprehensiveness of the search, the node to be searched is used as the new search start node until the search results do not include the node to be searched. In order to ensure the comprehensiveness of the located target node, the present application obtains the target search result corresponding to the alarm node based on the target node obtained in each search process. Therefore, the present application does not need to manually check the target node step by step, can improve the efficiency of searching the target node, and limit the search depth to ensure the search security.

[0014] In a possible embodiment, after searching the preceding key node of the search start node to determine the target node, the method further includes:

[0015] If the search duration of the current search process reaches a set time threshold, the search process is terminated and the search result is obtained.

[0016] In an embodiment of the present application, based on the relationship between the search duration of the current search process and the set time threshold, it is determined whether to terminate the search process. When the search duration of the current search process reaches the set time threshold, the search process is terminated and the search results are obtained, thereby limiting the search depth, reducing the occupied memory resources, reducing the occurrence of memory overflow, stack overflow and other problems, and improving search security.

[0017] In a possible embodiment, searching the preceding key node of the search start node to determine the target node includes:

[0018] During the search process, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completion state, and a running state.

[0019] In an embodiment of the present application, based on the state of the previous key node of the current search, it is determined whether the previous key node of the current search is the target node. There is no need to manually check the target node step by step, which can improve the efficiency of searching for the target node.

[0020] In a possible embodiment, judging whether the current searched previous key node is the target node based on the state of the current searched previous key node includes:

[0021] If the state of the previous key node currently being searched is an error state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped;

[0022] If the state of the previous key node currently being searched is a waiting state, and all the previous key nodes of the previous key node currently being searched are in a completed state, then the previous key node currently being searched is determined to be the target node, and all the previous key nodes of the previous key node currently being searched are skipped; if the state of the previous key node currently being searched is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the previous key node currently being searched, then it is determined that the previous key node currently being searched is not the target node;

[0023] If the state of the previous key node currently being searched is a completed state, determining that the previous key node currently being searched is not the target node, and skipping all previous key nodes of the previous key node currently being searched;

[0024] If the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

[0025] In an embodiment of the present application, based on the state of the previous key node of the current search, it is determined whether the previous key node of the current search is the target node, and a skipping strategy can be adopted for the previous key nodes that do not need to be searched, thereby further reducing the number of nodes that need to be searched and improving the efficiency of searching for the target node.

[0026] In a possible embodiment, judging whether the previous key node currently being searched is before the target node based on the state of the previous key node currently being searched further includes:

[0027] Make sure that there is no searched previous key node in the set list that matches the currently searched previous key node.

[0028] In an embodiment of the present application, in order to prevent the problem of a circular search causing the program to remain in a loop and unable to exit, it is determined that there is no searched previous key node in the set list that matches the current searched previous key node, to ensure that the current search is not a circular retrieval.

[0029] In a possible embodiment, the method further includes:

[0030] If there is a searched previous key node in the set list that matches the current searched previous key node, the current searched previous key node is skipped.

[0031] In an embodiment of the present application, if there is a searched previous key node in the set list that matches the previous key node of the current search, it indicates that this search is a circular search. Therefore, the previous key node of the current search is skipped to prevent the problem of a circular search causing the program to remain in the loop and unable to exit.

[0032] In a possible embodiment, after obtaining the target search result corresponding to the alarm node, the method further includes:

[0033] The target search result is saved in a cache, and a life cycle corresponding to the target search result is set;

[0034] If the storage time of the target search result reaches the life cycle, the target search result stored in the cache is deleted.

[0035] In an embodiment of the present application, the target search results are saved in a cache, and a life cycle corresponding to the target search results is set; if the storage time of the target search results reaches the life cycle, the target search results saved in the cache are deleted, thereby effectively saving the search results corresponding to each alarm node, facilitating user queries, thereby improving response speed and improving user experience.

[0036] In a possible embodiment, after setting the life cycle corresponding to the target search result, the method further includes:

[0037] In response to a query request for the search result of the alarm node triggered by a user, a target search result of the alarm node is determined from the cache based on the alarm node in the query request.

[0038] In an embodiment of the present application, in response to a query request for search results of an alarm node triggered by a user, a target search result of the alarm node is determined from a cache based on the alarm node in the query request, thereby increasing the response speed of the user request and improving the user experience.

[0039] In a second aspect, the present application provides a data processing device, the device comprising:

[0040] A detection module, used to detect an alarm node in a batch processing system, wherein the batch processing system includes a plurality of processing nodes, and the alarm node is a processing node that triggers an alarm condition among the plurality of processing nodes;

[0041] An execution module, the execution module includes a search module, a first judgment module and a second judgment module, and the execution module is used to use the alarm node as a search start node and repeatedly perform the following operations:

[0042] A search module is used to search for the preceding key node of the search starting node in a node network established based on the sequential relationship of the multiple processing nodes in the process of executing business processing, starting from the search starting node, to determine a target node, wherein the target node is the preceding key node that causes the search starting node to trigger an alarm condition;

[0043] The first judgment module is used to terminate the current search process and obtain the search result if the number of searched previous key nodes reaches a set threshold;

[0044] A second judgment module is used for, if the search result includes a node to be searched, using the node to be searched as a new search start node until the search result does not include the node to be searched, wherein the node to be searched is a previous key node that has not been searched in a target layer and a previous key node that is in a waiting state, and the target layer is a layer where the last searched previous key node is located or a layer below the layer;

[0045] The determination module is used to obtain the target search result corresponding to the alarm node based on the target node obtained in each search process.

[0046] In a possible embodiment, after searching the preceding key node of the search start node to determine the target node, the search module is further configured to:

[0047] If the search duration of the current search process reaches a set time threshold, the search process is terminated and the search result is obtained.

[0048] In a possible embodiment, the search module is used to:

[0049] During the search process, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completion state, and a running state.

[0050] In a possible embodiment, the search module is used to:

[0051] If the state of the previous key node currently being searched is an error state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped;

[0052] If the state of the previous key node currently being searched is a waiting state, and all the previous key nodes of the previous key node currently being searched are in a completed state, then the previous key node currently being searched is determined to be the target node, and all the previous key nodes of the previous key node currently being searched are skipped; if the state of the previous key node currently being searched is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the previous key node currently being searched, then it is determined that the previous key node currently being searched is not the target node;

[0053] If the state of the previous key node currently being searched is a completed state, determining that the previous key node currently being searched is not the target node, and skipping all previous key nodes of the previous key node currently being searched;

[0054] If the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

[0055] In a possible embodiment, before judging whether the previous key node currently being searched is the target node based on the state of the previous key node currently being searched, the search module is further configured to:

[0056] Make sure that there is no searched previous key node in the set list that matches the currently searched previous key node.

[0057] In a possible embodiment, the search module is further used for:

[0058] If there is a searched previous key node in the set list that matches the current searched previous key node, the current searched previous key node is skipped.

[0059] In a possible embodiment, after obtaining the target search result corresponding to the alarm node, the determination module is further used to:

[0060] The target search result is saved in a cache, and a life cycle corresponding to the target search result is set;

[0061] If the storage time of the target search result reaches the life cycle, the target search result stored in the cache is deleted.

[0062] In a possible embodiment, after setting the life cycle corresponding to the target search result, the determining module is further configured to:

[0063] In response to a query request for the search result of the alarm node triggered by a user, a target search result of the alarm node is determined from the cache based on the alarm node in the query request.

[0064] In a third aspect, the present application provides an electronic device, including:

[0065] A memory for storing program instructions;

[0066] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.

[0068] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0070] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0071] Figure 3 A detailed flow chart of the first data processing method provided in the embodiment of the present application;

[0072] Figure 4 A schematic diagram of a node network provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of another node network provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of a node network provided in an embodiment of the present application;

[0075] Figure 7 A detailed flow chart of a method for determining whether the previous key node currently being searched is a target node provided in an embodiment of the present application;

[0076] Figure 8 A detailed flow chart of the second data processing method provided in the embodiment of the present application;

[0077] Fig. 9 A structural diagram of a data processing device provided in an embodiment of the present application;

[0078] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0080] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0081] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0082] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0083] Before introducing the data processing method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0084] With the continuous development of computer technology, the use of computers for batch business processing has greatly improved the speed of business processing, saved manpower related to business processing, and has become the first choice for batch business processing in all walks of life.

[0085] As a post-data processing method, batch business processing has the characteristics of large concurrency, long time consumption and complicated processing logic. If the data volume increases, batch business errors and upstream system data supply delays occur, batch business delays (referred to as batch delays) may occur. In order to avoid batch delays affecting the availability of application online business, it is necessary to locate the cause of the delay in a timely manner, solve the problem and resume batch operation.

[0086] Currently, in most cases, the batch businesses being run are complex and intertwined. If an upstream task exception causes a delay in a downstream task, it is usually necessary to manually check each level to find the root cause, which is inefficient.

[0087] In order to solve the above problems, the present application proposes a data processing method, device, equipment and medium to improve the efficiency of searching for target nodes and limit the search depth to ensure search security.

[0088] First reference Figure 1 , which is a schematic diagram of an application scenario of an embodiment of the present application, including a collector 11 and a server 12. The collector 11 is used to collect node information of each processing node in the batch processing system and send it to the server 12; the server 12 is used to detect the alarm node in the batch processing system based on the node information of each processing node in the batch processing system sent by the collector 11, and search for the target node that causes the alarm node to trigger the alarm condition in the node network established based on the sequence relationship of multiple processing nodes in the process of executing business processing, and obtain the target search result corresponding to the alarm node.

[0089] In an embodiment of the present application, the server 12 detects the alarm node in the batch processing system based on the node information of each processing node in the batch processing system sent by the collector 11, wherein the batch processing system includes multiple processing nodes, and the alarm node is a processing node that triggers the alarm condition among the multiple processing nodes; the alarm node is used as the search start node, and the following operations are repeatedly performed: in a node network established based on the sequence relationship of multiple processing nodes in the execution of business processing, starting from the search start node, the previous key node of the search start node is searched to determine the target node, and the target node is the previous key node that causes the search start node to trigger the alarm condition; if the number of searched previous key nodes reaches a set threshold, the current search process is terminated to obtain the search result; if the search result includes the node to be searched, the node to be searched is used as the new search start node until the search result does not include the node to be searched, wherein the node to be searched is the previous key node that has not been searched in the target layer and the previous key node in the waiting state, and the target layer is the layer where the last searched previous key node is located or the next layer of the layer; based on the target node obtained in each search process, the target search result corresponding to the alarm node is obtained.

[0090] Reference below Figure 2 A flowchart of a data processing method is shown to illustrate the technical solution provided by the embodiment of the present application:

[0091] Step 201, detecting alarm nodes in a batch processing system;

[0092] The batch processing system includes multiple processing nodes, and the alarm node is a processing node that triggers an alarm condition among the multiple processing nodes. The alarm node refers to a node that triggers an alarm condition when the waiting time of the task to be executed by the processing node exceeds the set alarm time threshold. The alarm time threshold can be set according to actual conditions.

[0093] Step 202, taking the alarm node as the search start node, repeatedly performing the following operations: in a node network established based on the order relationship of multiple processing nodes in the process of executing business processing, starting from the search start node, searching for the previous key node of the search start node to determine the target node;

[0094] Among them, the target node is the preceding key node that causes the alarm condition of the search start node to be triggered. The preceding key node of the search start node includes the processing node whose processing flow is before the search start node. The sequence relationship of multiple processing nodes in the batch processing system in the process of executing business processing refers to the sequence processing relationship between the tasks to be processed by each processing node. For example, when the task to be processed by processing node A depends on the completion of the task to be processed by processing node B, then processing node B is before processing node A, and processing node B can be called the preceding key node or upstream node of processing node A.

[0095] Step 203: if the number of searched previous key nodes reaches a set threshold, the current search process is terminated and the search result is obtained;

[0096] Among them, the threshold value can be set according to the actual situation, for example, the threshold value can be set to 100. The search results can include only the nodes to be searched, or only the target nodes, or both the nodes to be searched and the target nodes. In addition to the above content, the search results can also include normal nodes. The present application can use set identifiers to mark the nodes to be searched, the target nodes and the normal nodes, wherein the set identifiers can be set according to the actual situation, for example, marking the nodes to be searched as C, marking the target nodes as Y, and marking the normal nodes as N. Therefore, the search results record information such as the node ID (Identification) and the identifier corresponding to the node.

[0097] In addition to the method in step 203 above, the present application can also limit the number of nodes searched each time by the following method: if the search duration of the current search process reaches a set time threshold, the search process is terminated to obtain the search results.

[0098] Among them, the setting time threshold can be set according to the actual situation, for example, the setting time threshold can be 1000ms (milliseconds). The search duration of the current search process is determined based on the alarm time of the alarm node and the search time corresponding to the current searched previous key node in the target layer. For example, the alarm time of the alarm node is 1:00:00, and the search time corresponding to the current searched previous key node in the target layer is 1:00:01, then the search duration of the current search process is 1 second.

[0099] Since in the prior art, after detecting the alarm node, the previous key node of the branch where the alarm node is located in the node network is retrieved, when the data volume of the current key node is large, a large amount of memory is required, and problems such as memory overflow and stack overflow may occur. Therefore, this application is based on the relationship between the number of searched previous key nodes and the set threshold, or based on the search duration of the current search process and the set time threshold, to limit the search depth, reduce the occupied memory resources, reduce the occurrence of problems such as memory overflow and stack overflow, and improve the search security.

[0100] Step 204: if the search result includes the node to be searched, the node to be searched is used as a new search start node until the search result does not include the node to be searched;

[0101] The nodes to be searched are the previous key nodes that have not been searched in the target layer and the previous key nodes in the waiting state, and the target layer is the layer where the last searched previous key node is located or the next layer of the layer. The search results may include one or more nodes to be searched.

[0102] Step 205: Based on the target node obtained in each search process, a target search result corresponding to the alarm node is obtained.

[0103] In an embodiment of the present application, after detecting the alarm node in the batch processing system, the alarm node is used as the search start node, and the following operations are repeatedly performed: in a node network established based on the sequential relationship of multiple processing nodes in the execution of business processing, the search start node is used as the starting point, and the front key node of the search start node is searched to determine the target node that causes the search start node to trigger the alarm condition; if the number of the searched front key nodes reaches the set threshold, the current search process is terminated, and the search results are obtained to limit the search depth, reduce the occupied memory resources, reduce the occurrence of problems such as memory overflow and stack overflow, improve the search security, and improve the efficiency of searching the target node; if the search results include the node to be searched, it means that the node network has not yet completed the complete retrieval, and some nodes need to be retrieved. Therefore, in order to ensure the comprehensiveness of the search, the node to be searched is used as the new search start node until the search results do not include the node to be searched. In order to ensure the comprehensiveness of the located target node, the present application obtains the target search result corresponding to the alarm node based on the target node obtained in each search process. Therefore, the present application does not need to manually check the target node step by step, can improve the efficiency of searching the target node, and limit the search depth to ensure the search security.

[0104] Since the methods of limiting the number of nodes to be searched each time are different, the data processing methods are also different. Therefore, for the method of limiting the number of nodes to be searched each time based on the number of searched previous key nodes and setting a threshold, Figure 3 A detailed flow chart of the first data processing method provided in the embodiment of the present application is as follows: Figure 3 As shown, at least the following steps 301-308 are included:

[0105] Step 301, detecting alarm nodes in a batch processing system.

[0106] Step 302: Use the alarm node as the search start node.

[0107] Step 303 , in a node network established based on the sequential relationship of multiple processing nodes in the process of executing business processing, starting from the search start node, searching for the preceding key node of the search start node to determine the target node.

[0108] Since in the prior art, a complete node network is established based on the sequence relationship between processing nodes in a batch processing system, after an alarm node is detected, the preceding key node of the branch where the alarm node is located in the node network is searched, and the complete node network occupies a large amount of memory, and problems such as memory overflow and stack overflow may occur. Therefore, for the embodiments of the present application, a node network is established based on the sequence relationship of multiple processing nodes in the process of executing business processing, including: based on the sequence relationship of multiple processing nodes in the process of executing business processing, a node network with the search start node as the root node is established.

[0109] For example, the alarm node is processing node A. The task to be processed by processing node A can only be performed after the tasks to be processed by processing node B and processing node C are completed. The task to be processed by processing node B can only be performed after the tasks to be processed by processing node D and processing node E are completed. The task to be processed by processing node C can only be performed after the task to be processed by processing node F is completed. Figure 4 A schematic diagram of a node network provided in an embodiment of the present application, such as Figure 4 As shown, based on the sequence of the six processing nodes in executing business processing, a node network with processing node A as the root node is established.

[0110] Since the use of the depth-first search algorithm for a million-level search may only find an abnormal point on a branch, but lose a large number of branches, and may cause the risk of OOM (Out of Memory) and SOF (Stack Over Flow) due to the data stack being too deep, the present application uses the breadth-first search (Breadth First Search, BFS) algorithm to search the previous key node of the search starting node to determine the target node. Among them, the BFS algorithm is a traversal algorithm for graphs and trees.

[0111] For example, Figure 4 As shown, in the process of searching the processing nodes in the node network using the BFS algorithm, processing node A is searched first, and then processing node B, processing node C, processing node D, processing node E and processing node F are searched in sequence.

[0112] Since a large number of query operations need to be performed on a large table during the search for the target node, a large number of IO (Input / Output) operations are generated, and the search is relatively time-consuming. Therefore, the present application uses a parallel method to search in the process of searching for the front key node of the current layer to solve the problem of long response time for large data volume searches, thereby improving the efficiency of searching for the target node. Among them, the parallel method can be a multi-threaded method, or other parallel methods. Specifically, when the current layer starts the parallel search for the front key node, the search for the next layer has not started. Only when the search for all the front key nodes of the current layer is completed, the front key nodes of the next layer will be searched; by blocking the multi-threading of the current layer, that is, the current layer will continue to search the next layer only when it meets the set conditions, thereby ensuring asynchrony but still searching according to the number of layers.

[0113] In the current data processing method, during the search process, each search operation needs to establish a connection with the database, and generally a long connection is used to maintain communication with the database. This data processing method occupies a large amount of database connection resources. The present application not only uses a long connection to maintain communication with the database, but also uses the foreach (loop) in Mybatis (a persistence layer framework) to form an ultra-long SQL (Structured Query Language) using the IN condition to reduce the occupation time of database connection resources. For example, there are 100 front key nodes in the 5th layer, and the prior art needs to query the database 100 times, while the present application uses 1 query to return all required results, thereby reducing the number of database connections and reducing database pressure.

[0114] Since the dependencies between processing nodes may be cross-domain, cross-component, and cross-batch, for such complex scenarios, this application defaults all dependencies to be cross-domain, cross-component, and cross-batch, reducing code if branches, simplifying the code and reducing the difficulty of code maintenance.

[0115] For example, in normal cases, the domain of processing node 1 is region A and the batch is 0001, and the domain of the previous key node of processing node 1 should be region A and the batch should be 0001. In special cases, the domain of processing node 2 is region A and the batch is 0001, and the domain of the previous key node of processing node 2 is region B and the batch is 0002. This application maintains all normal cases as special cases.

[0116] Step 304 , determine whether the number of searched previous key nodes reaches a set threshold, if so, execute step 305 , otherwise, execute step 303 .

[0117] Step 305, end the current search process and obtain the search results.

[0118] In the embodiment of the present application, the search results may include only the nodes to be searched, only the target nodes, or both the nodes to be searched and the target nodes. In addition to the above content, the search results may also include normal nodes.

[0119] For example, Figure 5 A schematic diagram of another node network provided in an embodiment of the present application, such as Figure 5 As shown, starting from node 1, node 2, node 3, node 4, node 5, node 6, node 7, node 8, node 9 and node 10 are searched in sequence. If the threshold is set to 2 and node 2 is the target node, after searching node 1 and node 2, it is determined that the number of searched front key nodes is 2, and the set threshold is reached, then the current search process is terminated to obtain the search result. The search result may include node 2 as the target node and node 3 as the node to be searched. The search result may also include node 1 as a normal node, node 2 as the target node, and node 3 as the node to be searched.

[0120] If the threshold is set to 3, and node 5 is the target node, then after searching nodes 1, 2, and 3, it is determined that the number of searched front-key nodes is 3, reaching the set threshold, then the current search process ends and the search result is obtained. If node 2 is in a waiting state, and nodes 4 and 5 are the front-key nodes of node 2, therefore, the search result can include nodes 2, 6, and 7 as nodes to be searched. The search result can also include nodes 1 and 3 as normal nodes, and nodes 2, 6, and 7 as nodes to be searched.

[0121] If the threshold is set to 2, and node 5 is the target node, then after searching node 1 and node 2, it is determined that node 2 is in a waiting state, and the number of searched front key nodes is 2, reaching the set threshold, then the current search process is terminated and the search result is obtained. The search result may include that node 2 and node 3 are both nodes to be searched. The search result may also include that node 1 is a normal node, and node 2 and node 3 are both nodes to be searched.

[0122] If the threshold is set to 10, and node 6 is the target node, then after searching the 10 nodes in sequence, it is determined that the number of searched front key nodes is 10 and reaches the set threshold, then the current search process is terminated to obtain the search results. The search results may include node 6 as the target node. The search results may also include node 1, node 2, node 3, node 4, node 5, node 7, node 8, node 9 and node 10 are all normal nodes, and node 6 is the target node.

[0123] If the threshold is set to 10 and there is no target node, then after searching the 10 nodes in turn, it is determined that the number of searched front key nodes is 10 and reaches the set threshold, then the current search process is ended and the search results are obtained. The search results include node 1, node 2, node 3, node 4, node 5, node 6, node 7, node 8, node 9 and node 10 are all normal nodes.

[0124] Step 306, determine whether the search result includes the node to be searched, if so, execute step 307, otherwise execute step 308.

[0125] Step 307: Use the node to be searched as a new search start node.

[0126] In the embodiment of the present application, the search result may include one or more nodes to be searched. When the search result may include multiple nodes to be searched, each node to be searched needs to be used as a new search start node.

[0127] For example, the search results include two nodes to be searched, namely node 1 and node 2. Node 1 needs to be used as a new search start node first, and step 303 needs to be executed. Then, node 2 needs to be used as a new search start node, and step 303 needs to be executed.

[0128] Step 308, based on the target node obtained in each search process, obtain the target search result corresponding to the alarm node.

[0129] The target search result includes information such as the alarm node ID and the target node ID.

[0130] For example, in the process of searching for the target node of alarm node 1, there are 4 retrieval processes in total. The target node obtained in the first retrieval process is node 3, the target node obtained in the second retrieval process is node 7, the target node is not obtained in the third retrieval process, and the target nodes obtained in the fourth retrieval process are nodes 16 and 18. Therefore, based on the target nodes obtained in each search process, the target search results corresponding to the alarm node are obtained, and the target search results include node 1 as the alarm node, and nodes 3, 7, 16, and 18 as the target nodes.

[0131] In the embodiment of the present application, after obtaining the target search result corresponding to the alarm node, the method further includes:

[0132] The target search results are saved in the cache, and the life cycle corresponding to the target search results is set; if the storage time of the target search results reaches the life cycle, the target search results saved in the cache are deleted.

[0133] Among them, the cache can be Redis (a data structure storage system). Redis is an open source, memory-based data structure storage system that can be used as a database, cache, and message middleware. The life cycle corresponding to the target search result set above can be set according to actual conditions. For example, the life cycle can be 1 minute. The search results of different alarm nodes can be set with the same life cycle or different life cycles. The storage duration of the target search result is determined based on the time when the target search result is saved in the cache and the current time. For example, if the search result is saved in the cache at 1:00:02 and the current time is 1:00:05, the storage duration of the search result is 3 seconds.

[0134] In order to improve the response speed of user requests, reduce database pressure and reduce the large number of database connections generated by impatient users' repeated clicks on the front-end page, after setting the life cycle corresponding to the target search results, it also includes: responding to the query request for the search results of the alarm node triggered by the user, and determining the target search results of the alarm node from the cache based on the alarm node in the query request.

[0135] The query request includes relevant information of the alarm node, for example, the alarm node ID. If the target search result of the alarm node can be determined from the cache based on the alarm node in the query request, it indicates that the target search result of the alarm node has not been deleted from the cache. If the target search result of the alarm node cannot be determined from the cache based on the alarm node in the query request, it indicates that the target search result of the alarm node has been deleted from the cache, or the target search result of the alarm node has not been obtained.

[0136] Optionally, searching the preceding key node of the search start node to determine the target node includes:

[0137] During the search process, based on the state of the current searched previous key node, it is determined whether the current searched previous key node is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completed state, and a running state.

[0138] In order to prevent the problem of circular search causing the program to be unable to exit the loop, the present application also includes, before judging whether the current searched front key node is the target node based on the state of the current searched front key node: determining that there is no searched front key node in the set list that matches the current searched front key node.

[0139] The information of the searched previous key node is saved in the setting list, and the information of the searched previous key node may include the searched previous key node ID. For example, if node 4 needs to be searched currently, node 1, node 2 and node 3 are saved in the setting list. Before searching for node 4, it is determined that node 4 does not exist in the setting list. Therefore, it is determined that this search is not a circular search, and node 4 is searched.

[0140] In the embodiment of the present application, if there is a searched previous key node in the setting list that matches the current searched previous key node, the current searched previous key node is skipped.

[0141] For example, if node 4 needs to be searched currently, node 1, node 2, node 3 and node 4 are saved in the setting list. Before searching for node 4, it is determined that node 4 exists in the setting list. Therefore, it is determined that this search is a circular search and node 4 is not searched, that is, node 4 is skipped.

[0142] In the embodiment of the present application, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, including the following five situations:

[0143] In the first case, if the state of the current searched previous key node is an error state, the current searched previous key node is determined to be the target node, and all previous key nodes of the current searched previous key node are skipped.

[0144] Specifically, if the current searched front key node is in an error state, it indicates that the current searched front key node is the reason that causes the alarm node to trigger the alarm condition. Therefore, the current searched front key node is registered as the target node, and all the front key nodes of the current searched front key node are skipped.

[0145] For example, Figure 6 A schematic diagram of a node network provided in an embodiment of the present application, such as Figure 6 As shown, starting from node 1, node 2, node 3, node 4, node 5, node 6, node 7, node 8, node 9, node 10, node 11, node 12, node 13, node 14 and node 15 are searched in sequence. If the threshold is set to 6, and the state of the currently searched node 2 is an error state, node 2 is determined to be the target node, and all the previous key nodes of node 2 are skipped. The previous key nodes of node 2 include node 5, node 6, node 10, node 11 and node 12. Therefore, it is necessary to search node 3, node 4, node 7 and node 8 in sequence. At this time, the number of the searched previous key nodes has reached 6, and the current search process is terminated to obtain the search results.

[0146] In the second case, if the state of the current searched previous key node is in the waiting state, and all previous key nodes of the current searched previous key node are in the completed state, the current searched previous key node is determined to be the target node, and all previous key nodes of the current searched previous key node are skipped.

[0147] Specifically, if the state of the front key node of the current search is in a waiting state, since the reason why the front key node of the current search is in the waiting state may be caused by its front key node or by the front key node of the current search itself, the states of all the front key nodes of the front key node of the current search are counted. When all the front key nodes of the front key node of the current search are in the completed state, it indicates that the front key node of the current search is in the waiting state due to the front key node of the current search itself. The front key node of the current search is registered as the target node, and all the front key nodes of the front key node of the current search are skipped.

[0148] For example, Figure 6 As shown, if the threshold is set to 6, and the status of the currently searched node 2 is in the waiting state, then it is necessary to search nodes 3, 4, 5 and 6 in sequence. If it is determined that all the predecessor nodes of node 2, that is, nodes 5 and 6 are in the completed state, then it means that the predecessor nodes of node 5, that is, nodes 10 and 11, are also in the completed state, and the predecessor node of node 6, that is, node 12, is also in the completed state. Therefore, node 2 is the target node, and the number of predecessor nodes that have been searched has reached 6, ending the current search process and obtaining the search results.

[0149] In the third case, if the state of the current searched previous key node is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the current searched previous key node, it is determined that the current searched previous key node is not a target node.

[0150] Specifically, if the status of the front key node currently being searched is in a waiting state, and all the front key nodes of the front key node currently being searched are in a completed state, it indicates that the front key node currently being searched is in a waiting state due to the front key node of the front key node currently being searched, and it is determined that the front key node currently being searched is not the target node.

[0151] For example, Figure 6 As shown, if the threshold is set to 6, and the status of the currently searched node 2 is in the waiting state, then it is necessary to search nodes 3, 4, 5 and 6 in turn. If it is determined that node 6 is not in the completed state among all the previous key nodes of node 2, it means that node 2 is not the target node. At this time, the number of previous key nodes searched has reached 6, and the current search process is ended to obtain the search result.

[0152] In an embodiment of the present application, when the status of the front key node currently being searched is in a waiting state, since the number of front key nodes that have been searched has reached a set threshold, only the status of some of the front key nodes of the front key node currently being searched can be counted. At this time, it cannot be determined whether the front key node currently being searched is the target node. Therefore, the front key node currently being searched is added to the search results as a node to be searched.

[0153] For example, Figure 6 As shown in the figure, if the threshold is set to 5, and the state of the currently searched node 2 is in the waiting state, it is necessary to search nodes 3, 4, and 5 in sequence to determine that node 5 is in the completed state. Since the number of searched front key nodes has reached 5 at this time, the current search process is terminated to obtain the search result. Since node 6 cannot be searched, it is impossible to determine whether node 2 is the target node, so node 2 is added to the search result as a node to be searched.

[0154] In the fourth case, if the state of the previous key node currently being searched is a completed state, it is determined that the previous key node currently being searched is not a target node, and all previous key nodes of the previous key node currently being searched are skipped.

[0155] Specifically, if the status of the current searched previous key node is the completed status, it indicates that the current searched previous key node is a normal node. Since all the previous key nodes of the current searched previous key node are in the completed status, the current searched previous key node can be in the completed status. Therefore, all the previous key nodes of the current searched previous key node are also normal nodes. It is determined that the current searched previous key node is not the target node, and all the previous key nodes of the current searched previous key node are skipped.

[0156] For example, Figure 6 As shown, if the threshold is set to 8, and the state of the currently searched node 2 is completed, it is determined that node 2 is not the target node, and all the previous key nodes of node 2 are skipped, that is, the previous key nodes of node 2 are not searched, and the previous key nodes of node 2 include node 5, node 6, node 10, node 11 and node 12. Therefore, it is necessary to search nodes 3, node 4, node 7, node 8, node 9 and node 13 in sequence. At this time, the number of searched previous key nodes has reached 8, and the current search process is terminated to obtain the search result.

[0157] In the fifth case, if the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

[0158] Specifically, if the status of the front key node currently being searched is in the running state, it indicates that the front key node currently being searched is the reason that causes the alarm node to trigger the alarm condition. Since all the front key nodes of the front key node currently being searched are in the completed state, the front key node currently being searched can be in the running state. Therefore, all the front key nodes of the front key node currently being searched are normal nodes, the front key node currently being searched is registered as the target node, and all the front key nodes of the front key node currently being searched are skipped.

[0159] For example, Figure 6 As shown, if the threshold is set to 8 and the state of node 3 is in the running state, node 3 is determined to be the target node, and all the previous key nodes of node 3 are skipped, that is, the previous key nodes of node 3 are not searched, and the previous key nodes of node 3 include node 7, node 13 and node 14. Therefore, it is necessary to search node 4, node 5, node 6, node 8 and node 9 in sequence. At this time, the number of searched previous key nodes has reached 8, and the current search process is ended to obtain the search result.

[0160] Figure 7 A detailed flow chart of a method for determining whether the current searched previous key node is a target node provided in an embodiment of the present application, such as Figure 7 As shown, at least the following steps 701-308 are included:

[0161] Step 701, obtain the status of the previous key node currently being searched.

[0162] Step 702, determine whether the state of the current searched front key node is an error state, if so, execute step 703, otherwise execute step 704;

[0163] Step 703: determine that the previous key node currently being searched is the target node, and skip all previous key nodes of the previous key node currently being searched.

[0164] Step 704, determine whether the previous key node currently being searched is in a waiting state, if so, execute step 705, otherwise, execute step 708.

[0165] Step 705 , determining whether all the previous key nodes of the currently searched previous key node are in a completed state, if so, executing step 706 , otherwise, executing step 707 .

[0166] Step 706, determine that the previous key node currently being searched is the target node, and skip all previous key nodes of the previous key node currently being searched.

[0167] Step 707, determine that the previous key node currently being searched is not the target node.

[0168] Step 708, determine whether the previous key node of the current search is in a completed state, if so, execute step 709, otherwise, execute step 710.

[0169] Step 709: determine that the previous key node currently being searched is not the target node, and skip all previous key nodes of the previous key node currently being searched.

[0170] Step 710, determine that the state of the current searched previous key node is in the running state, determine that the current searched previous key node is the target node, and skip all previous key nodes of the current searched previous key node.

[0171] In an embodiment of the present application, during the search process, whether the current searched previous key node is the target node is determined based on the state of the current searched previous key node, and a skipping strategy can be adopted for the previous key nodes that do not need to be searched, thereby reducing the number of nodes that need to be searched and improving the efficiency of searching for the target node.

[0172] Regarding the method of limiting the number of nodes searched each time based on the search duration and setting a threshold, Figure 8 A detailed flow chart of the second data processing method provided in the embodiment of the present application is as follows: Figure 8 As shown, at least the following steps 801-808 are included:

[0173] Step 801, detecting alarm nodes in a batch processing system.

[0174] Step 802: Use the alarm node as the search start node.

[0175] Step 803 , in a node network established based on the sequential relationship of multiple processing nodes in the process of executing business processing, starting from the search start node, searching for the preceding key node of the search start node to determine the target node.

[0176] The specific implementation process of the above step 803 can refer to the specific implementation process of the above step 303, which will not be repeated here.

[0177] Step 804, determine whether the search duration of the current search process reaches the set time threshold, if so, execute step 805, otherwise, execute step 803.

[0178] Step 805, end the current search process and obtain the search results.

[0179] Step 806, determine whether the search result includes the node to be searched, if so, execute step 807, otherwise execute step 808.

[0180] Step 807: Use the node to be searched as a new search start node.

[0181] Step 808, based on the target node obtained in each search process, obtain the target search result corresponding to the alarm node.

[0182] Based on the same inventive concept, the present application embodiment provides a data processing device, please refer to Fig. 9 , the device comprises:

[0183] A detection module 91, configured to detect an alarm node in a batch processing system, wherein the batch processing system includes a plurality of processing nodes, and the alarm node is a processing node that triggers an alarm condition among the plurality of processing nodes;

[0184] The execution module 92 includes a search module 921, a first judgment module 922, and a second judgment module 923. The execution module 92 is used to use the alarm node as a search start node and repeatedly perform the following operations:

[0185] A search module 921 is used to search for the preceding key node of the search starting node in a node network established based on the sequential relationship of the multiple processing nodes in the process of executing business processing, starting from the search starting node, to determine a target node, wherein the target node is the preceding key node that causes the search starting node to trigger an alarm condition;

[0186] The first judgment module 922 is used to terminate the current search process and obtain the search result if the number of searched previous key nodes reaches a set threshold;

[0187] The second judgment module 923 is used for, if the search result includes a node to be searched, using the node to be searched as a new search start node until the search result does not include the node to be searched, wherein the node to be searched is a previous key node that has not been searched in the target layer and a previous key node that is in a waiting state, and the target layer is the layer where the last searched previous key node is located or the next layer of the layer;

[0188] The determination module 93 is used to obtain the target search result corresponding to the alarm node based on the target node obtained in each search process.

[0189] As an optional implementation manner, after searching the preceding key node of the search start node to determine the target node, the search module 921 is further used to:

[0190] If the search duration of the current search process reaches a set time threshold, the search process is terminated and the search result is obtained.

[0191] As an optional implementation, the search module 921 is used to:

[0192] During the search process, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completion state, and a running state.

[0193] As an optional implementation, the search module 921 is used to:

[0194] If the state of the previous key node currently being searched is an error state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped;

[0195] If the state of the previous key node currently being searched is a waiting state, and all the previous key nodes of the previous key node currently being searched are in a completed state, then the previous key node currently being searched is determined to be the target node, and all the previous key nodes of the previous key node currently being searched are skipped; if the state of the previous key node currently being searched is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the previous key node currently being searched, then it is determined that the previous key node currently being searched is not the target node;

[0196] If the state of the previous key node currently being searched is a completed state, determining that the previous key node currently being searched is not the target node, and skipping all previous key nodes of the previous key node currently being searched;

[0197] If the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

[0198] As an optional implementation, based on the state of the previous key node currently being searched, judging whether the previous key node currently being searched is before the target node, the search module 921 is further used to:

[0199] Make sure that there is no searched previous key node in the set list that matches the currently searched previous key node.

[0200] As an optional implementation, the search module 921 is further configured to:

[0201] If there is a searched previous key node in the set list that matches the current searched previous key node, the current searched previous key node is skipped.

[0202] As an optional implementation manner, after obtaining the target search result corresponding to the alarm node, the determination module 93 is further used to:

[0203] The target search result is saved in a cache, and a life cycle corresponding to the target search result is set;

[0204] If the storage time of the target search result reaches the life cycle, the target search result stored in the cache is deleted.

[0205] As an optional implementation, after setting the life cycle corresponding to the target search result, the determination module 93 is further used to:

[0206] In response to a query request for the search result of the alarm node triggered by a user, a target search result of the alarm node is determined from the cache based on the alarm node in the query request.

[0207] Based on the same inventive concept, the present application embodiment provides an electronic device, which can realize the functions of the data processing device discussed above. Please refer to Fig.10 , the device comprises a processor 101 and a memory 102, wherein the memory 102 is used to store program instructions;

[0208] The processor 101 calls the program instructions stored in the memory and executes the program instructions to implement the following steps:

[0209] Detecting an alarm node in a batch processing system, wherein the batch processing system includes a plurality of processing nodes, and the alarm node is a processing node that triggers an alarm condition among the plurality of processing nodes;

[0210] Take the alarm node as the search start node and repeat the following operations:

[0211] In a node network established based on the sequential relationship of the multiple processing nodes in the process of executing business processing, starting from the search start node, searching for the previous key node of the search start node to determine a target node, the target node being the previous key node that causes the search start node to trigger an alarm condition;

[0212] If the number of searched front key nodes reaches the set threshold, the current search process ends and the search result is obtained;

[0213] If the search result includes a node to be searched, the node to be searched is used as a new search start node until the search result does not include the node to be searched, wherein the node to be searched is a previous key node that has not been searched in the target layer and a previous key node that is in a waiting state, and the target layer is the layer where the last searched previous key node is located or the next layer of the layer;

[0214] Based on the target node obtained in each search process, a target search result corresponding to the alarm node is obtained.

[0215] As an optional implementation manner, after searching the preceding key node of the search start node to determine the target node, the processor 101 is further configured to execute:

[0216] If the search duration of the current search process reaches a set time threshold, the search process is terminated and the search result is obtained.

[0217] As an optional implementation manner, searching the preceding key node of the search start node to determine the target node includes:

[0218] During the search process, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completion state, and a running state.

[0219] As an optional implementation manner, judging whether the current searched previous key node is the target node based on the state of the current searched previous key node includes:

[0220] If the state of the previous key node currently being searched is an error state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped;

[0221] If the state of the previous key node currently being searched is a waiting state, and all the previous key nodes of the previous key node currently being searched are in a completed state, then the previous key node currently being searched is determined to be the target node, and all the previous key nodes of the previous key node currently being searched are skipped; if the state of the previous key node currently being searched is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the previous key node currently being searched, then it is determined that the previous key node currently being searched is not the target node;

[0222] If the state of the previous key node currently being searched is a completed state, determining that the previous key node currently being searched is not the target node, and skipping all previous key nodes of the previous key node currently being searched;

[0223] If the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

[0224] As an optional implementation manner, before determining whether the current searched previous key node is the target node based on the state of the current searched previous key node, the processor 101 is further configured to execute:

[0225] Make sure that there is no searched previous key node in the set list that matches the currently searched previous key node.

[0226] As an optional implementation manner, the processor 101 is further configured to execute:

[0227] If there is a searched previous key node in the set list that matches the current searched previous key node, the current searched previous key node is skipped.

[0228] As an optional implementation manner, after obtaining the target search result corresponding to the alarm node, the processor 101 is further configured to execute:

[0229] The target search result is saved in a cache, and a life cycle corresponding to the target search result is set;

[0230] If the storage time of the target search result reaches the life cycle, the target search result stored in the cache is deleted.

[0231] As an optional implementation manner, after setting the life cycle corresponding to the target search result, the processor 101 is further configured to execute:

[0232] In response to a query request for the search result of the alarm node triggered by a user, a target search result of the alarm node is determined from the cache based on the alarm node in the query request.

[0233] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data processing method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0234] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data processing method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0235] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0236] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0237] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0238] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0239] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method includes: Detecting an alarm node in a batch processing system, wherein the batch processing system includes a plurality of processing nodes, and the alarm node is a processing node that triggers an alarm condition among the plurality of processing nodes; Take the alarm node as the search start node and repeat the following operations: In a node network established based on the sequential relationship of the multiple processing nodes in the process of executing business processing, starting from the search start node, searching for the previous key node of the search start node to determine a target node, the target node being the previous key node that causes the search start node to trigger an alarm condition; If the number of searched previous key nodes reaches the set threshold, the current search process ends and the search result is obtained; If the search result includes a node to be searched, the node to be searched is used as a new search start node until the search result does not include the node to be searched, wherein the node to be searched is a previous key node that has not been searched in the target layer and a previous key node that is in a waiting state, and the target layer is the layer where the last searched previous key node is located or the next layer of the layer; Based on the target node obtained in each search process, a target search result corresponding to the alarm node is obtained.

2. The method according to claim 1, characterized in that After searching the preceding key node of the search start node to determine the target node, the method further includes: If the search duration of the current search process reaches a set time threshold, the search process is terminated and the search result is obtained.

3. The method according to claim 1, characterized in that The step of searching the preceding key node of the search start node to determine the target node includes: During the search process, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completion state, and a running state.

4. The method according to claim 3, characterized in that The determining whether the current searched previous key node is the target node based on the state of the current searched previous key node includes: If the state of the previous key node currently being searched is an error state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped; If the state of the previous key node currently being searched is a waiting state, and all the previous key nodes of the previous key node currently being searched are in a completed state, then the previous key node currently being searched is determined to be the target node, and all the previous key nodes of the previous key node currently being searched are skipped; if the state of the previous key node currently being searched is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the previous key node currently being searched, then it is determined that the previous key node currently being searched is not the target node; If the state of the previous key node currently being searched is a completed state, determining that the previous key node currently being searched is not the target node, and skipping all previous key nodes of the previous key node currently being searched; If the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

5. The method according to claim 3, characterized in that The determining, based on the state of the previous key node currently being searched, whether the previous key node currently being searched is before the target node, further includes: Make sure that there is no searched previous key node in the set list that matches the currently searched previous key node.

6. The method according to claim 5, characterized in that The method further comprises: If there is a searched previous key node in the set list that matches the current searched previous key node, the current searched previous key node is skipped.

7. The method according to claim 1, characterized in that After obtaining the target search result corresponding to the alarm node, the method further includes: The target search result is saved in a cache, and a life cycle corresponding to the target search result is set; If the storage time of the target search result reaches the life cycle, the target search result stored in the cache is deleted.

8. The method according to claim 7, characterized in that After setting the life cycle corresponding to the target search result, the method further includes: In response to a query request for the search result of the alarm node triggered by a user, a target search result of the alarm node is determined from the cache based on the alarm node in the query request.

9. A data processing device, characterized in that: The device includes: A detection module, used to detect an alarm node in a batch processing system, wherein the batch processing system includes a plurality of processing nodes, and the alarm node is a processing node that triggers an alarm condition among the plurality of processing nodes; An execution module, the execution module includes a search module, a first judgment module and a second judgment module, and the execution module is used to use the alarm node as a search start node and repeatedly perform the following operations: A search module is used to search for the preceding key node of the search starting node in a node network established based on the sequential relationship of the multiple processing nodes in the process of executing business processing, starting from the search starting node, to determine a target node, wherein the target node is the preceding key node that causes the search starting node to trigger an alarm condition; The first judgment module is used to terminate the current search process and obtain the search result if the number of searched previous key nodes reaches a set threshold; A second judgment module is used for, if the search result includes a node to be searched, using the node to be searched as a new search start node until the search result does not include the node to be searched, wherein the node to be searched is a previous key node that has not been searched in a target layer and a previous key node that is in a waiting state, and the target layer is a layer where the last searched previous key node is located or a layer below the layer; The determination module is used to obtain the target search result corresponding to the alarm node based on the target node obtained in each search process.

10. The device according to claim 9, characterized in that After searching the preceding key node of the search start node to determine the target node, the search module is further used to: If the search duration of the current search process reaches a set time threshold, the search process is terminated and the search result is obtained.

11. The device according to claim 9, characterized in that The search module is used to: During the search process, based on the state of the previous key node currently being searched, it is determined whether the previous key node currently being searched is the target node, wherein the state of the previous key node includes one of an error state, a waiting state, a completion state, and a running state.

12. The device according to claim 11, characterized in that The search module is used to: If the state of the previous key node currently being searched is an error state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped; If the state of the previous key node currently being searched is a waiting state, and all the previous key nodes of the previous key node currently being searched are in a completed state, then the previous key node currently being searched is determined to be the target node, and all the previous key nodes of the previous key node currently being searched are skipped; if the state of the previous key node currently being searched is a waiting state, and there is a previous key node that is not in a completed state among all the previous key nodes of the previous key node currently being searched, then it is determined that the previous key node currently being searched is not the target node; If the state of the previous key node currently being searched is a completed state, determining that the previous key node currently being searched is not the target node, and skipping all previous key nodes of the previous key node currently being searched; If the state of the previous key node currently being searched is in the running state, the previous key node currently being searched is determined to be the target node, and all previous key nodes of the previous key node currently being searched are skipped.

13. The device according to claim 11, characterized in that Based on the state of the previous key node currently being searched, judging whether the previous key node currently being searched is before the target node, the search module is further used to: Make sure that there is no searched previous key node in the set list that matches the currently searched previous key node.

14. The device according to claim 13, characterized in that The search module is also used to: If there is a searched previous key node in the set list that matches the current searched previous key node, the current searched previous key node is skipped.

15. The device according to claim 9, characterized in that After obtaining the target search result corresponding to the alarm node, the determination module is further used to: The target search result is saved in a cache, and a life cycle corresponding to the target search result is set; If the storage time of the target search result reaches the life cycle, the target search result stored in the cache is deleted.

16. The device according to claim 15, characterized in that After setting the life cycle corresponding to the target search result, the determination module is further used to: In response to a query request for the search result of the alarm node triggered by a user, a target search result of the alarm node is determined from the cache based on the alarm node in the query request.

17. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods of claims 1-8 according to the obtained program instructions.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images