Data processing link tracking method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data, and discloses a data processing link tracking method, device and equipment and a storage medium, the method comprises the following steps: determining a target system, and running a cleaning task based on a predetermined data cleaning tool; after the task runs at the key node, running state information in the log cache is obtained, and the running state information is reported to the step recorder; when the task runs, detecting the data based on a first detection rule in the data cleaning tool, and after successful running of the task is finished, if wrong first data is detected, reporting the first data to a data error collector; the running state information is sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running state information and the first data are displayed on the application layer. By means of the mode, the data cleaning links can be effectively tracked, abnormal points can be rapidly positioned, and error data can be checked.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and specifically to a data processing link tracking method, device, equipment and storage medium. Background Art

[0002] As the business develops, the data scale and business scenarios are constantly expanding, and a large amount of data needs to be cleaned to obtain the professional data required by the enterprise-level data warehouse.

[0003] In the existing data warehouse data cleaning work, there are data errors with different probabilities, so it is necessary to track the data cleaning link to find the cause of the anomaly. Existing data cleaning tools generally focus on data cleaning. For highly complex and long-term data cleaning processes, they rely on the experience of developers in locating anomalies and troubleshooting errors, and cannot effectively track the data cleaning link and troubleshoot anomalies. Summary of the invention

[0004] In view of the above problems, an embodiment of the present invention provides a data processing link tracking method, device, equipment and storage medium to solve the problems existing in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a data processing link tracking method is provided, comprising:

[0006] Determine a target system, and run a task of cleaning data in the target system based on a predetermined data cleaning tool;

[0007] When the task is running and after the task is finished, the first reporting step and the second reporting step are performed:

[0008] The first reporting step includes, when the task is completed at a predetermined key node, obtaining the running status information in the log cache, and reporting the running status information to a preset step recorder, wherein the running status information includes at least normal running information and / or abnormal running information;

[0009] The second reporting step includes, when the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully run, if erroneous first data is detected, reporting the first data to a preset data error collector;

[0010] The running status information is sent to an application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer.

[0011] In an optional manner, the method further includes:

[0012] The cleaned data is stored in a first storage layer of the database, and the cleaned data in the first storage layer is detected based on a second detection rule preset in the data error collector;

[0013] If erroneous second data is detected, the second data is sent to the application layer based on the data error collector, and the second data is displayed at the application layer.

[0014] In an optional manner, the method further includes: storing the running status information, the first data and the second data persistently in a second storage layer of the database.

[0015] In an optional manner, if the running status information includes the running exception information, the first reporting step further includes, when the task is completed at a predetermined key node, obtaining the stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder;

[0016] The sending of the running status information to the application layer based on the step recorder, sending of the first data to the application layer based on the data error collector, and displaying of the running status information and the first data on the application layer further includes:

[0017] The running status information and the stack information are sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information, the stack information and the first data are displayed on the application layer.

[0018] In an optional manner, the key nodes include a first key node, a second key node, and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node, and a task initialization node, and the display of the running status information on the application layer includes:

[0019] At the application layer, the abnormal operation information is prompted according to the prompt priority corresponding to the key node to which the abnormal operation information belongs.

[0020] In an alternative approach,

[0021] The first reporting step also includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtaining the running status information from the message queue;

[0022] The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

[0023] In an optional manner, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an indication of no data error to the data error collector.

[0024] According to another aspect of an embodiment of the present invention, a data processing link tracking device is provided, comprising:

[0025] An operation module is used to determine a target system and run a task of cleaning data in the target system based on a predetermined data cleaning tool;

[0026] The reporting module is used to trigger the first reporting unit and the second reporting unit when the task is running and after the running is completed:

[0027] The first reporting unit is used to obtain the running status information in the log cache when the task is completed at the predetermined key node, and report the running status information to the preset step recorder, wherein the running status information at least includes normal running information and / or abnormal running information;

[0028] The second reporting unit is used to detect the data based on the first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, report the first data to a preset data error collector;

[0029] A display module is used to send the running status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the running status information and the first data on the application layer.

[0030] According to another aspect of an embodiment of the present invention, there is provided a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method described above.

[0031] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and when the executable instruction is executed on a computer device, the computer device executes the method described above.

[0032] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the running status information in the log cache is obtained, and the running status information is reported to a preset step recorder, and the running status information at least includes normal running information and / or abnormal running information; when the task is running, the data is detected based on a predetermined first detection rule in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the running status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer. This statement embodiment uses the running status information of key nodes to check the task execution status and locate abnormal points. Through the first data, data problems such as data omissions, data anomalies, and data inconsistencies can be checked, so that the link of data cleaning can be effectively tracked and abnormal points can be quickly located, and erroneous data can be checked.

[0033] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0035] Figure 1 A schematic diagram of the system architecture of a data processing link tracking method provided by an embodiment of the present invention is shown;

[0036] Figure 2 A schematic diagram showing a flow chart of a data processing link tracking method provided by an embodiment of the present invention;

[0037] Figure 3 The figure shows the flow chart of the data cleaning tool kettle running the cleaning processing task;

[0038] Figure 4 A schematic diagram showing the structure of a data processing link tracking device provided by an embodiment of the present invention is shown;

[0039] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0041] Figure 1 A schematic diagram of the system architecture of the data processing link tracking method provided by an embodiment of the present invention is shown. Figure 1 As shown, the system architecture is the architecture of the data cleaning tool, including the application layer, the scheduling layer, the task operation layer and the storage layer. The task operation layer includes a step recorder and a data error collector. Among them, the application layer is mainly used for task management, and the reported operation status information and error data can be viewed. The scheduling layer is used for task scheduling and real-time monitoring of the operation status of tasks. The task operation layer includes a task operation cluster, a step recorder and a data error collector. During the task operation, all events that occur are recorded in the log cache, and the information of related events is reported to the step recorder, so that the operation status information of key nodes can be synchronized to the step recorder. The data error collector collects error data for tasks that run successfully. Among them, the data cleaning tool pre-adds detection logic for business scenarios. According to the detection logic during the task operation, the error data can be detected, and the error data will be reported to the data error collector for collection when the task exits. Finally, the step recorder and the data error collector will asynchronously persist the operation status information of the key nodes and the detected error data to the storage layer and display them in the application layer. The storage layer is used to store management data such as the operation status information of key nodes, error data polarization, and other execution information. Through this system architecture, the data cleaning link can be tracked based on the operation status information and error data, and the abnormal points can be quickly located.

[0042] Figure 2 FIG. 1 is a flow chart showing a data processing link tracking method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0043] Step 101 , determining a target system, and running a task of cleaning data in the target system based on a predetermined data cleaning tool.

[0044] Before performing data cleaning, the target system for data cleaning must be clearly defined. The requirements for data cleaning are different for different target systems, different business scenarios, and different analysis needs. In this embodiment, the data cleaning tool kettle is used for data cleaning. The data cleaning tool kettle is an optimized data cleaning tool, which includes a step recorder and a data error collector. It has the function of collecting operation status information and error data. The data cleaning link can be traced according to the operation status information and error data, so as to quickly locate abnormal points in highly complex and long data processing links. Of course, other optimized data cleaning tools that can achieve the above functions can also be used for data cleaning, and there is no excessive limitation here.

[0045] Step 102, when the task is running and after running, execute the first reporting step and the second reporting step, wherein the first reporting step includes, when the task is completed at a predetermined key node, obtaining the running status information in the log cache, and reporting the running status information to a preset step recorder, wherein the running status information at least includes normal running information and / or abnormal running information.

[0046] In this embodiment, the running status information is reported when the task is running and before exiting the task after the running. Among them, this embodiment predetermines multiple key nodes that are important in the data cleaning process, and multiple key nodes can form the entire link of data cleaning. During the running of the task, the running status of the key nodes is monitored. The log cache of the database caches all events during the running period, but it will be cleared after the task ends and exits. Therefore, when the running of the key node ends, this embodiment will obtain the corresponding running status information from the log cache of the database, and then report the running status information to the step recorder so that the step recorder can record it.

[0047] Optionally, the above-mentioned first reporting step also includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtains the running status information from the message queue.

[0048] In this embodiment, the execution of the task is completed by a process. The process is a dynamic execution process of a program with independent functions on a certain data set, and is the basic unit for resource allocation and scheduling of the operating system. The process not only includes the instructions and data of the data cleaning program, but also includes all registers of the program counter and the processor and the process stack for storing temporary data. The kernel stores the process in a bidirectional circular linked list (i.e., a process linked list). Each item in the linked list is a process control block task-struct. Task_struct is a descriptor of the process, which contains the unique identifier, affinity, task status, authority, operation statistics, scheduling information, signal processing, memory management, file system and other information of the process. After obtaining the operation status information stored in the log cache logbuffer, in order not to affect the operation of the task, this embodiment creates an asynchronous first thread, and pushes the operation status information to a predetermined message queue through the asynchronous first thread. The step recorder, as a message consumer, consumes directly from the message queue to obtain the operation status information.

[0049] In one embodiment, if the running status information includes the running exception information, the first reporting step also includes, when the task is completed at a predetermined key node, obtaining the stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder.

[0050] Among them, the stack information Stack Trace is a set of information that records the order of function calls. In this embodiment, detailed stack information is generated when the data cleaning program throws an exception. The stack information can display the call path when the data cleaning program is executed to a certain point. By reporting the stack information to the step recorder for recording and analyzing the stack information, it is basically possible to obtain program logic errors and locate corresponding exception problems, thereby improving the efficiency and accuracy of data exception analysis.

[0051] The second reporting step includes, when the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, reporting the first data to a preset data error collector.

[0052] In this embodiment, the data cleaning tool pre-adds detection logic for business scenarios, namely the first detection rule. During the task operation, the first data with errors can be detected according to the first detection rule. The first data includes the component name of the data with errors, the data primary key, the cause of the data error, etc., and the task is marked as a data error state. After the task is successfully completed, the data with errors will be centrally processed and temporarily stored in the cache, and then reported to the data error collector in a centralized manner so that the data error collector can record it.

[0053] Optionally, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

[0054] In this embodiment, after the task has successfully completed, if an erroneous first data is detected, in order not to affect the operation of the task, this embodiment creates an asynchronous second thread, and pushes the first data to the message queue through the asynchronous second thread. The data error collector acts as a message consumer and directly consumes from the message queue to obtain the first data, thereby obtaining the erroneous first data.

[0055] Optionally, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an indication of no data error to the data error collector.

[0056] Among them, after the task is successfully completed, if no erroneous first data is detected, that is, all the data are correct, an identification of no data error is generated, and an asynchronous second thread is created. The identification of no data error is pushed to the message queue through the asynchronous second thread. The data error collector, as a message consumer, directly consumes from the message queue and obtains the identification of no data error, and then reports the identification of no data error to the data error collector to complete the data cleaning closed loop. The development and operation personnel can clearly understand the data cleaning situation through this identification.

[0057] Step 103: Send the running status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the running status information and the first data on the application layer.

[0058] In this embodiment, the step recorder persists the operation status information to the storage layer in an asynchronous manner and sends it to the application layer, and the data error collector also persists the first data to the storage layer in an asynchronous manner and sends it to the application layer. Optionally, the step recorder can also summarize the problems according to the operation status information, so that the development and operation personnel can review the problems through the error code provided by the operation exception information. In the management page of the application layer, the operation status information and the first data are displayed. The development and operation personnel can view the task execution status, the abnormal points and the abnormal reasons when the operation is abnormal through the operation status information of the displayed key nodes. Through the first data, data problems such as data omissions, data anomalies and data inconsistencies can be found, so that the link of data cleaning can be effectively tracked and the abnormal points can be quickly located, and the erroneous data can be checked. In this embodiment, according to the above-mentioned data cleaning method, an enterprise-level task management application platform can be integrated, and data cleaning task management, task monitoring, abnormal location and error data troubleshooting can be completed through the visual page, reducing manual dependence and reducing operation and maintenance costs.

[0059] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a predetermined first detection rule in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. This declaration embodiment can effectively track the link of data cleaning and quickly locate abnormal points, and troubleshoot erroneous data.

[0060] Optionally, if the running status information includes running exception information, and the stack information corresponding to the running exception information is also reported to the step recorder, the running status information is sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer. It also includes: sending the running status information and the stack information to the application layer based on the step recorder, sending the first data to the application layer based on the data error collector, and displaying the running status information, the stack information and the first data on the application layer.

[0061] In this embodiment, the application layer displays the stack information while displaying the running status information and the first data. By displaying the data link of data cleaning through the running status information, the abnormal point position can be quickly located, and the detailed stack information corresponding to the abnormality when it occurs can be intuitively viewed.

[0062] In one embodiment, the key nodes include a first key node, a second key node and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node and a task initialization node, and the display of the running status information on the application layer includes: on the application layer, prompting the running abnormality information according to the prompt priority corresponding to the key node to which the running abnormality information belongs.

[0063] In this embodiment, for the task running node with the highest prompt priority, if an operation abnormality occurs, the highest level of prompt is given to the operation abnormality information. The highest level of prompt method may be a combination of one or more, for example, displaying the operation abnormality information in red, or displaying it in red, flashing, and sounding an alarm. For the first data acquisition node and reporting node with the second highest prompt priority, if an operation abnormality occurs, the next highest level of prompt is given to the operation abnormality information. The next highest level of prompt method may also be a combination of one or more, for example, displaying the operation abnormality information in yellow. For the task configuration loading node, Meta configuration loading node, database table loading node, data event merging node, and task initialization node with the lowest prompt priority, if an operation abnormality occurs, the lowest level of prompt is given to the operation abnormality information, for example, displaying the operation abnormality information in blue.

[0064] Among them, combined with reference Figure 3 Taking the data cleaning tool kettle as an example, the data cleaning task executes the task configuration loading node, Meta configuration loading node, database table loading node, data event merging node, task initialization node, kettle task running node, first data acquisition node and first data reporting node in the order of key node execution, and finally reports the running status information to the step recorder, and reports the first erroneous data to the data error collector.

[0065] Among them, in the task configuration loading node, its normal operation information is CONFIG (loading task configuration status);

[0066] In the Meta configuration loading node, by detecting the kettle file, you can get its normal operation information as LOADING (loading kettle file meta configuration status);

[0067] In the database table loading node, the normal operation information is FETCHING (database table data loading event status);

[0068] At the data event merge node, the primary table and the secondary table are merged, and the normal operation information is FETCHED (merged data event status);

[0069] At the task initialization node, the normal operation information is INITIALIZED (Kettle task initialization status);

[0070] In the kettle task running node, it includes subdivided sub-nodes, including the start component (i.e. kettle), waiting for the component to end and the end component. After the start component, it also includes monitoring interrupt signals, abnormal exit processing and end component. In the start component, its normal operation information is STARTING (Run Kettle Task-Start Component Status), and in the end component, its normal operation information is FINISHED (Run Kettle Task-End Component Status);

[0071] At the first data acquisition node (i.e., acquiring data processing results), its normal operation information includes AGGREGATING (status of starting to acquire data processing results) and AGGREGATED (status of completing acquiring data processing results);

[0072] At the first data reporting node (ie, reporting the data processing result), its normal operation information includes PUSHING (status of starting to report the data processing result) and PUSHED (status of completing reporting the data processing result).

[0073] In addition, in the database table loading node, the operation exception information is EVENT_ERR (database table data event loading failure status), which means that an exception occurs during the processing of the data table data event;

[0074] At the task initialization node, the running exception information is INIT_ERR (Kettle task initialization failure status), which means that an exception occurs during the task initialization process, such as database connection pool creation exception, middleware connectivity test exception, and storage bucket non-existence.

[0075] In the sub-node end component, its operation exception information is EXEC_ERR (Run Kettle task-operation failure status), which means that any exception occurs during the task execution, such as null pointer exception, SQL execution timeout exception, and source file does not exist;

[0076] At the first data reporting node, its operation exception information includes RPC_ERR (data processing result reporting failure status), which indicates that an exception occurs during asynchronous communication with the data error collector.

[0077] In one embodiment, the method further includes: storing the cleansed data in a first storage layer of a database, and detecting the cleansed data in the first storage layer based on a second detection rule preset in the data error collector; if erroneous second data is detected, sending the second data to the application layer based on the data error collector, and displaying the second data at the application layer.

[0078] In this embodiment, the cleaned data is stored in the first storage layer of the database. Optionally, the first storage layer is an operational data store (ODS). The ODS layer can provide the cleaned data to the data warehouse and data mart, and then further organize and analyze the data. In this embodiment, a second detection rule is also preset in the data error collector. Different from the first detection rule based on business logic mentioned above, the second detection rule can be a detection rule obtained by machine learning training based on big data, or it can also be a set of universal and basic detection rules based on the experience of development experts. The second detection rule can be used to perform further in-depth detection on the cleaned data. If an erroneous second data is detected, the second data is sent to the application layer for display to improve the accuracy of data cleaning.

[0079] In one embodiment, the method further includes: storing the running status information, the first data, and the second data persistently in a second storage layer of the database.

[0080] In this embodiment, the operation status information, the first data and the second data are stored persistently in the second storage layer of the database. The second storage layer can be a data warehouse layer (DWH). The data warehouse layer stores data that has been fully processed and integrated. These data have undergone processes such as cleaning and conversion to facilitate subsequent global analysis and decision support.

[0081] Figure 4 FIG. 2 shows a schematic diagram of the structure of a data processing link tracking device according to an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0082] An operation module 10 is used to determine a target system and to run a task of cleaning data in the target system based on a predetermined data cleaning tool;

[0083] The reporting module 20 is used to trigger the first reporting unit 201 and the second reporting unit 202 when the task is running and after the task is finished:

[0084] The first reporting unit 201 is used to obtain the running status information in the log cache when the task is completed at a predetermined key node, and report the running status information to a preset step recorder, wherein the running status information at least includes normal running information and / or abnormal running information;

[0085] A second reporting unit 202 is used to detect the data based on the first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, report the first data to a preset data error collector;

[0086] The display module 30 is used to send the running status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the running status information and the first data on the application layer.

[0087] Among them, the embodiment of the data processing link tracking device is basically the same as the above-mentioned data processing link tracking method.

[0088] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a predetermined first detection rule in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. This declaration embodiment can effectively track the link of data cleaning and quickly locate abnormal points, and troubleshoot erroneous data.

[0089] Figure 5 The schematic diagram of the structure of the computer device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0090] like Figure 5 As shown, the computer device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0091] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other computer devices such as client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned computer device embodiment.

[0092] Specifically, the program 410 may include program code including computer executable instructions.

[0093] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0094] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0095] The program 410 may be specifically called by the processor 402 to enable the computer device to perform the following operations:

[0096] Determine a target system, and run a task of cleaning data in the target system based on a predetermined data cleaning tool;

[0097] When the task is running and after the task is finished, the first reporting step and the second reporting step are performed:

[0098] The first reporting step includes, when the task is completed at a predetermined key node, obtaining the running status information in the log cache, and reporting the running status information to a preset step recorder, wherein the running status information includes at least normal running information and / or abnormal running information;

[0099] The second reporting step includes, when the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully run, if erroneous first data is detected, reporting the first data to a preset data error collector;

[0100] The running status information is sent to an application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer.

[0101] In an optional manner, the method further includes:

[0102] The cleaned data is stored in a first storage layer of the database, and the cleaned data in the first storage layer is detected based on a second detection rule preset in the data error collector;

[0103] If erroneous second data is detected, the second data is sent to the application layer based on the data error collector, and the second data is displayed at the application layer.

[0104] In an optional manner, the method further includes: storing the running status information, the first data and the second data persistently in a second storage layer of the database.

[0105] In an optional manner, if the running status information includes the running exception information, the first reporting step further includes, when the task is completed at a predetermined key node, obtaining the stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder;

[0106] The sending of the running status information to the application layer based on the step recorder, sending of the first data to the application layer based on the data error collector, and displaying of the running status information and the first data on the application layer further includes:

[0107] The running status information and the stack information are sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information, the stack information and the first data are displayed on the application layer.

[0108] In an optional manner, the key nodes include a first key node, a second key node, and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node, and a task initialization node, and the display of the running status information on the application layer includes:

[0109] At the application layer, the abnormal operation information is prompted according to the prompt priority corresponding to the key node to which the abnormal operation information belongs.

[0110] In an alternative approach,

[0111] The first reporting step also includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtaining the running status information from the message queue;

[0112] The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

[0113] In an optional manner, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an indication of no data error to the data error collector.

[0114] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a predetermined first detection rule in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. This declaration embodiment can effectively track the link of data cleaning and quickly locate abnormal points, and troubleshoot erroneous data.

[0115] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device, the computer device executes any of the above method embodiments.

[0116] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computer device to execute any of the above method embodiments.

[0117] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer is enabled to execute any of the above method embodiments.

[0118] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description made to specific languages ​​above is for disclosing the best mode of the present invention.

[0119] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0120] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly recited in each claim.

[0121] Those skilled in the art will appreciate that the modules in the computer device in the embodiment can be adaptively changed and set in one or more computer devices different from the embodiment. The modules or units or components in the embodiment can be combined into one module or unit or component, and they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or computer device disclosed in this manner can be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0122] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data processing link tracking method, characterized in that: The method comprises: Determine a target system, and run a task of cleaning data in the target system based on a predetermined data cleaning tool; When the task is running and after the task is finished, the first reporting step and the second reporting step are performed: The first reporting step includes, when the task is completed at a predetermined key node, obtaining the running status information in the log cache, and reporting the running status information to a preset step recorder, wherein the running status information includes at least normal running information and / or abnormal running information; The second reporting step includes, when the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully run, if erroneous first data is detected, reporting the first data to a preset data error collector; The running status information is sent to an application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer.

2. The method according to claim 1, characterized in that The method further comprises: The cleaned data is stored in a first storage layer of the database, and the cleaned data in the first storage layer is detected based on a second detection rule preset in the data error collector; If erroneous second data is detected, the second data is sent to the application layer based on the data error collector, and the second data is displayed at the application layer.

3. The method according to claim 2, characterized in that The method further includes: storing the running status information, the first data, and the second data persistently in a second storage layer of the database.

4. The method according to claim 1, characterized in that: If the running status information includes the running exception information, the first reporting step further includes, when the task is completed at a predetermined key node, obtaining stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder; The sending of the running status information to the application layer based on the step recorder, sending of the first data to the application layer based on the data error collector, and displaying of the running status information and the first data on the application layer further includes: The running status information and the stack information are sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information, the stack information and the first data are displayed on the application layer.

5. The method according to claim 4, characterized in that The key nodes include a first key node, a second key node and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node and a task initialization node, and the display of the running status information on the application layer includes: At the application layer, the abnormal operation information is prompted according to the prompt priority corresponding to the key node to which the abnormal operation information belongs.

6. The method according to claim 1, characterized in that The first reporting step also includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtaining the running status information from the message queue; The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

7. The method according to claim 1, characterized in that The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an indication of no data error to the data error collector.

8. A data processing link tracking device, characterized in that: include: An operation module is used to determine a target system and run a task of cleaning data in the target system based on a predetermined data cleaning tool; The reporting module is used to trigger the first reporting unit and the second reporting unit when the task is running and after the running is completed: The first reporting unit is used to obtain the running status information in the log cache when the task is completed at the predetermined key node, and report the running status information to the preset step recorder, wherein the running status information at least includes normal running information and / or abnormal running information; The second reporting unit is used to detect the data based on the first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, report the first data to a preset data error collector; A display module is used to send the running status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the running status information and the first data on the application layer.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device, the computer device executes the method according to any one of claims 1 to 7.

Images