Data processing link tracking method, device, equipment and storage medium

Abstract

The present invention relates to the field of data technology, and discloses a data processing link tracking method, device, equipment and storage medium, the method comprising: determining a target system, running a cleaning task based on a predetermined data cleaning tool; when the task is completed at a key node, obtaining the running status information in the log cache, and reporting the running status information to a step recorder; while the task is running, detecting the data based on a first detection rule in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, reporting the first data to a data error collector; sending the running status information to the application layer based on the step recorder, sending the first data to the application layer based on the data error collector, and displaying the running status information and the first data on the application layer. Through the above-mentioned method, the embodiment of the present invention can effectively track the data cleaning link and quickly locate abnormal points, and troubleshoot erroneous data.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data processing link tracking method, apparatus, device, and storage medium. Background Art

[0002] As the business develops, the data scale and business scenarios are constantly expanding, and a large amount of data needs to be cleaned to obtain the professional data required for enterprise-level data warehouses.

[0003] Existing data warehouse data cleaning processes often have varying probabilities of data errors, necessitating the need to track the data cleaning chain to identify the causes of these anomalies. Existing data cleaning tools typically focus on data cleaning. For highly complex and extensive data cleaning processes, these tools rely on the developer's experience to locate anomalies and troubleshoot errors, making them ineffective in effectively tracking the data cleaning chain and troubleshooting anomalies. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a data processing link tracking method, apparatus, device and storage medium to solve the problems existing in the prior art.

[0005] According to one aspect of an embodiment of the present invention, a data processing link tracing method is provided, comprising:

[0006] Determine the target system and run a task of cleaning the data in the target system based on a predetermined data cleaning tool;

[0007] When the task is running and after the task is finished, the first reporting step and the second reporting step are performed:

[0008] The first reporting step includes, when the task is completed at a predetermined key node, obtaining operation status information in a log cache, and reporting the operation status information to a preset step recorder, wherein the operation status information includes at least normal operation information and / or abnormal operation information;

[0009] The second reporting step includes, when the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, reporting the first data to a preset data error collector;

[0010] The running status information is sent to an application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer.

[0011] In an optional manner, the method further includes:

[0012] storing the cleaned data in a first storage layer of a database, and detecting the cleaned data in the first storage layer based on a second detection rule preset in the data error collector;

[0013] If erroneous second data is detected, the second data is sent to the application layer based on the data error collector, and the second data is displayed at the application layer.

[0014] In an optional manner, the method further includes: storing the running status information, the first data, and the second data persistently in a second storage layer of the database.

[0015] In an optional manner, if the running status information includes the running exception information, the first reporting step further includes, when the task is completed at a predetermined key node, obtaining stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder;

[0016] The step recorder is used to send the running status information to the application layer, the data error collector is used to send the first data to the application layer, and the running status information and the first data are displayed on the application layer, further comprising:

[0017] The running status information and the stack information are sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information, the stack information and the first data are displayed on the application layer.

[0018] In an optional manner, the key nodes include a first key node, a second key node, and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node, and a task initialization node, and the display of the running status information on the application layer includes:

[0019] At the application layer, the abnormal operation information is prompted according to the prompt priority corresponding to the key node to which the abnormal operation information belongs.

[0020] In an alternative approach,

[0021] The first reporting step further includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtaining the running status information from the message queue;

[0022] The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

[0023] In an optional manner, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an identifier of no data error to the data error collector.

[0024] According to another aspect of an embodiment of the present invention, a data processing link tracking device is provided, comprising:

[0025] An operation module is used to determine a target system and run a task of cleaning data in the target system based on a predetermined data cleaning tool;

[0026] The reporting module is used to trigger the first reporting unit and the second reporting unit when the task is running and after the running is completed:

[0027] The first reporting unit is configured to obtain operation status information in the log cache when the task is completed at a predetermined key node, and report the operation status information to a preset step recorder, wherein the operation status information includes at least normal operation information and / or abnormal operation information;

[0028] The second reporting unit is configured to detect the data based on a first detection rule predetermined in the data cleaning tool, and report the first data to a preset data error collector if an erroneous first data is detected after the task is successfully completed;

[0029] A display module is used to send the operating status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the operating status information and the first data on the application layer.

[0030] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method described above.

[0031] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device, the computer device executes the method described above.

[0032] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a predetermined first detection rule in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. The embodiment of this statement uses the operation status information of the key nodes to view the task execution status and locate abnormal points. Through the first data, data problems such as data omissions, data anomalies and data inconsistencies can be found, so that the data cleaning link can be effectively tracked and the abnormal points can be quickly located, and erroneous data can be checked.

[0033] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0035] Figure 1 A schematic diagram of the system architecture of a data processing link tracking method provided by an embodiment of the present invention is shown;

[0036] Figure 2 A schematic diagram showing a flow chart of a data processing link tracking method provided by an embodiment of the present invention is shown;

[0037] Figure 3 This figure shows a flow chart of the data cleaning tool kettle running the cleaning process task;

[0038] Figure 4 A schematic structural diagram of a data processing link tracking device provided by an embodiment of the present invention is shown;

[0039] Figure 5 A schematic structural diagram of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0041] Figure 1 A schematic diagram of the system architecture of the data processing link tracking method provided by an embodiment of the present invention is shown. Figure 1 The system architecture shown in the figure is that of a data cleaning tool, consisting of an application layer, a scheduling layer, a task execution layer, and a storage layer. The task execution layer includes a step recorder and a data error collector. The application layer primarily manages tasks and allows viewing reported execution status information and error data. The scheduling layer schedules tasks and monitors the execution status of tasks in real time. The task execution layer includes a task execution cluster, a step recorder, and a data error collector. During task execution, all events occurring are recorded in the log buffer and relevant event information is reported to the step recorder, enabling synchronization of the execution status information of key nodes with the step recorder. The data error collector collects error data from successfully executed tasks. The data cleaning tool has pre-implemented detection logic tailored to business scenarios. During task execution, this detection logic detects error data and reports it to the data error collector upon task exit. Finally, the step recorder and data error collector asynchronously persist the execution status information of key nodes and detected error data to the storage layer for display at the application layer. The storage layer is used to store management data such as the operating status of key nodes, error data, and other execution information. This system architecture allows tracking of data cleaning links based on operating status information and error data, and quickly locates anomalies.

[0042] Figure 2 FIG. 1 shows a flow chart of a data processing link tracking method according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0043] Step 101 : determining a target system and running a task of cleaning data in the target system based on a predetermined data cleaning tool.

[0044] Before performing data cleaning, it is necessary to first clarify the target system for data cleaning. Different target systems, different business scenarios, and different analysis needs have different requirements for data cleaning. In this embodiment, the data cleaning tool kettle is used for data cleaning. The data cleaning tool kettle is an optimized data cleaning tool that includes a step recorder and a data error collector. It has the function of collecting running status information and error data. According to the running status information and error data, the data cleaning link can be traced so as to quickly locate abnormal points in highly complex and long data processing links. Of course, other optimized data cleaning tools that can achieve the above functions can also be used for data cleaning, and there is no excessive limitation here.

[0045] Step 102, when the task is running and after running, execute the first reporting step and the second reporting step, wherein the first reporting step includes, when the task is completed at a predetermined key node, obtaining the running status information in the log cache, and reporting the running status information to a preset step recorder, wherein the running status information at least includes normal running information and / or abnormal running information.

[0046] In this embodiment, the running status information is reported while the task is running and before exiting the task after the running. In particular, this embodiment predetermines multiple key nodes that are important in the data cleaning process. Multiple key nodes can form the entire link of data cleaning. During the running of the task, the running status of the key nodes is monitored. The log cache of the database caches all events during the running period, but will be cleared after the task ends and exits. Therefore, when the key node ends running, this embodiment will obtain the corresponding running status information from the log cache of the database, and then report the running status information to the step recorder so that the step recorder can record it.

[0047] Optionally, the above-mentioned first reporting step also includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtains the running status information from the message queue.

[0048] In this embodiment, the execution of the task is completed by a process. A process is a dynamic execution process of a program with independent functions on a certain data set, and is the basic unit for resource allocation and scheduling of the operating system. The process not only includes the instructions and data of the data cleaning program, but also includes all registers of the program counter and the processor and the process stack for storing temporary data. The kernel stores the process in a bidirectional circular linked list (i.e., a process linked list). Each item in the linked list is a process control block task-struct. Task_struct is a descriptor of the process, which includes the unique identifier, affinity, task status, permissions, operation statistics, scheduling information, signal processing, memory management, file system and other information of the process. After obtaining the operation status information stored in the log cache logbuffer, in order not to affect the operation of the task, this embodiment creates an asynchronous first thread, and pushes the operation status information to a predetermined message queue through the asynchronous first thread. The step recorder, as a message consumer, consumes directly from the message queue to obtain the operation status information.

[0049] In one embodiment, if the running status information includes the running exception information, the first reporting step also includes, when the task is completed at a predetermined key node, obtaining the stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder.

[0050] Among them, the stack information Stack Trace is a set of information that records the sequence of function calls. In this embodiment, detailed stack information is generated when the data cleaning program throws an exception. The stack information can show the call path when the data cleaning program is executed to a certain point. By reporting the stack information to the step recorder for recording and analyzing the stack information, it is basically possible to obtain program logic errors and locate corresponding exception problems, thereby improving the efficiency and accuracy of data exception analysis.

[0051] The second reporting step includes, when the task is running, detecting the data based on the first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, reporting the first data to a preset data error collector.

[0052] In this embodiment, the data cleaning tool pre-implements business scenario-specific detection logic, known as the first detection rule. During task execution, this first detection rule detects the first data error. The first data includes the component name, primary key, and cause of the error, and identifies the task as in a data error state. After the task successfully completes, the error data is centrally processed and temporarily cached. It is then reported to the data error collector for logging.

[0053] Optionally, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

[0054] In this embodiment, after the task is successfully completed, if an erroneous first data is detected, in order not to affect the operation of the task, this embodiment creates an asynchronous second thread, and pushes the first data to the message queue through the asynchronous second thread. The data error collector acts as a message consumer and consumes directly from the message queue to obtain the first data, thereby obtaining the erroneous first data.

[0055] Optionally, the second reporting step also includes, during the running of the task, detecting the data based on the first detection rule in the data cleaning tool; after the task is successfully completed, if no erroneous first data is detected, reporting an indication of no data error to the data error collector.

[0056] Among them, after the task is successfully completed, if no erroneous first data is detected, that is, all the data are correct, an identifier of no data error is generated, and an asynchronous second thread is created. The identifier of no data error is pushed to the message queue through the asynchronous second thread. The data error collector, as a message consumer, consumes directly from the message queue and obtains the identifier of no data error, and then reports the identifier of no data error to the data error collector to complete the data cleaning closed loop. Development and operation personnel can clearly understand the data cleaning situation through this identifier.

[0057] Step 103: Send the running status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the running status information and the first data on the application layer.

[0058] In this embodiment, the step recorder asynchronously persists the operation status information to the storage layer and sends it to the application layer, and the data error collector also asynchronously persists the first data to the storage layer and sends it to the application layer. Optionally, the step recorder can also summarize problems based on the operation status information so that development and operation personnel can review the problems through the error codes provided by the operation exception information. In the management page of the application layer, the operation status information and the first data are displayed. Development and operation personnel can view the task execution status, abnormal points and causes of abnormalities during operation exceptions through the displayed operation status information of the key nodes. Through the first data, data problems such as data omissions, data anomalies and data inconsistencies can be found. Therefore, the data cleaning link can be effectively tracked and the abnormal points can be quickly located, and erroneous data can be checked. This embodiment can integrate an enterprise-level task management application platform based on the above-mentioned data cleaning method, and complete data cleaning task management, task monitoring, abnormal location and error data troubleshooting through a visual page, reducing manual dependence and reducing operation and maintenance costs.

[0059] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. This embodiment of the declaration can effectively track the link of data cleaning and quickly locate abnormal points, and check erroneous data.

[0060] Optionally, if the running status information includes running exception information, and the stack information corresponding to the running exception information is also reported to the step recorder, then the running status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer. It also includes: sending the running status information and the stack information to the application layer based on the step recorder, sending the first data to the application layer based on the data error collector, and displaying the running status information, the stack information and the first data on the application layer.

[0061] In this embodiment, the application layer displays the stack information while displaying the running status information and the first data. By displaying the data link of data cleaning through the running status information, the abnormal point position can be quickly located, and the detailed stack information corresponding to the abnormality can be intuitively viewed.

[0062] In one embodiment, the key nodes include a first key node, a second key node and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node and a task initialization node, and the display of the running status information on the application layer includes: on the application layer, prompting the running exception information according to the prompt priority corresponding to the key node to which the running exception information belongs.

[0063] In this embodiment, for the task running node with the highest prompt priority, if an operation abnormality occurs, the highest level prompt is given to the operation abnormality information. The highest level prompt method may be a combination of one or more methods, for example, displaying the operation abnormality information in red, or displaying it in red, flashing, and using an alarm sound. For the first data acquisition node and reporting node with the second highest prompt priority, if an operation abnormality occurs, the second highest level prompt is given to the operation abnormality information. The second highest level prompt method may also be a combination of one or more methods, for example, displaying the operation abnormality information in yellow. For the task configuration loading node, Meta configuration loading node, database table loading node, data event merging node, and task initialization node with the lowest prompt priority, if an operation abnormality occurs, the lowest level prompt is given to the operation abnormality information, for example, displaying the operation abnormality information in blue.

[0064] Among them, refer to Figure 3 Taking the data cleaning tool kettle as an example, the data cleaning task executes the task configuration loading node, Meta configuration loading node, database table loading node, data event merging node, task initialization node, kettle task running node, first data acquisition node and first data reporting node in the order of key node execution, and finally reports the running status information to the step recorder and reports the first erroneous data to the data error collector.

[0065] Among them, in the task configuration loading node, its normal operation information is CONFIG (loading task configuration status);

[0066] In the Meta configuration loading node, by detecting the kettle file, you can get its normal operation information as LOADING (loading kettle file meta configuration status);

[0067] In the database table loading node, the normal operation information is FETCHING (loading database table data event status);

[0068] At the data event merge node, the primary table and the secondary table are merged, and the normal operation information is FETCHED (merge data event status);

[0069] At the task initialization node, its normal operation information is INITIALIZED (Kettle task initialization status);

[0070] In the kettle task running node, it includes subdivided sub-nodes, including the starting component (i.e., kettle), waiting for the component to end, and the ending component. After the starting component, it also includes monitoring interrupt signals, abnormal exit processing, and ending components. In the starting component, its normal operation information is STARTING (Running Kettle Task-Starting Component Status), and in the ending component, its normal operation information is FINISHED (Running Kettle Task-Ending Component Status);

[0071] At the first data acquisition node (i.e., acquiring data processing results), its normal operation information includes AGGREGATING (status of starting to acquire data processing results) and AGGREGATED (status of completing acquiring data processing results);

[0072] At the first data reporting node (ie, reporting the data processing result), its normal operation information includes PUSHING (status of starting to report the data processing result) and PUSHED (status of completing reporting the data processing result).

[0073] In addition, in the database table loading node, the operation exception information is EVENT_ERR (database table data loading event failure status), which means that an exception occurs during the processing of the data table data event;

[0074] At the task initialization node, the running exception information is INIT_ERR (Kettle task initialization failure status), which means that an exception occurred during the task initialization process, such as database connection pool creation exception, middleware connectivity test exception, and bucket non-existence.

[0075] In the sub-node end component, its operation exception information is EXEC_ERR (Run Kettle task - Operation failure status), which means any exception occurs during the task execution, such as null pointer exception, SQL execution timeout exception, and source file does not exist;

[0076] At the first data reporting node, the operation exception information includes RPC_ERR (data processing result reporting failure status), which indicates that an exception occurs during asynchronous communication with the data error collector.

[0077] In one embodiment, the method further includes: storing the cleaned data in a first storage layer of a database, and detecting the cleaned data in the first storage layer based on a second detection rule preset in the data error collector; if erroneous second data is detected, sending the second data to the application layer based on the data error collector, and displaying the second data at the application layer.

[0078] In this embodiment, the cleaned data is stored in the first storage layer of the database. Optionally, the first storage layer is the operational data store (ODS). The ODS layer can provide the cleaned data to the data warehouse and data mart, and then further organize and analyze the data. In this embodiment, a second detection rule is also preset in the data error collector. Different from the first detection rule based on business logic mentioned above, the second detection rule can be a detection rule obtained by machine learning training based on big data, or it can also be a set of general and basic detection rules based on the experience of development experts. The second detection rule can be used to perform further in-depth detection on the cleaned data. If an erroneous second data is detected, the second data is sent to the application layer for display to improve the accuracy of data cleaning.

[0079] In one embodiment, the method further includes: storing the running status information, the first data, and the second data persistently in a second storage layer of the database.

[0080] In this embodiment, the operation status information, the first data and the second data are persistently stored in the second storage layer of the database. The second storage layer can be a data warehouse layer (DWH). The data warehouse layer stores data that has been fully processed and integrated. These data have undergone processes such as cleaning and conversion to facilitate subsequent global analysis and decision support.

[0081] Figure 4 FIG. 1 shows a schematic diagram of the structure of a data processing link tracking device according to an embodiment of the present invention. Figure 4 As shown, the device includes:

[0082] An operation module 10 is used to determine a target system and run a task of cleaning data in the target system based on a predetermined data cleaning tool;

[0083] The reporting module 20 is configured to trigger the first reporting unit 201 and the second reporting unit 202 during and after the task is executed:

[0084] The first reporting unit 201 is configured to obtain operation status information in the log cache when the task is completed at a predetermined key node, and report the operation status information to a preset step recorder, wherein the operation status information includes at least normal operation information and / or abnormal operation information;

[0085] A second reporting unit 202 is configured to detect data based on a first detection rule predetermined in the data cleaning tool, and report the first data to a preset data error collector if an erroneous first data is detected after the task is successfully completed;

[0086] The display module 30 is used to send the running status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the running status information and the first data on the application layer.

[0087] Among them, the embodiment of the data processing link tracking device is basically the same as the above-mentioned data processing link tracking method.

[0088] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. This embodiment of the declaration can effectively track the link of data cleaning and quickly locate abnormal points, and check erroneous data.

[0089] Figure 5 The schematic diagram of the structure of the computer device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0090] like Figure 5 As shown, the computer device may include: a processor 402 , a communications interface 404 , a memory 406 , and a communication bus 408 .

[0091] Processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other computer devices, such as client devices or other server network elements. Processor 402 is used to execute program 410, which may specifically perform the steps described above for the computer device embodiment.

[0092] Specifically, the program 410 may include program code including computer-executable instructions.

[0093] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a computer device may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs.

[0094] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0095] The program 410 may be specifically called by the processor 402 to cause the computer device to perform the following operations:

[0096] Determine the target system and run a task of cleaning the data in the target system based on a predetermined data cleaning tool;

[0097] When the task is running and after the task is finished, the first reporting step and the second reporting step are performed:

[0098] The first reporting step includes, when the task is completed at a predetermined key node, obtaining operation status information in a log cache, and reporting the operation status information to a preset step recorder, wherein the operation status information includes at least normal operation information and / or abnormal operation information;

[0099] The second reporting step includes, while the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, reporting the first data to a preset data error collector;

[0100] The running status information is sent to an application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer.

[0101] In an optional manner, the method further includes:

[0102] storing the cleaned data in a first storage layer of a database, and detecting the cleaned data in the first storage layer based on a second detection rule preset in the data error collector;

[0103] If erroneous second data is detected, the second data is sent to the application layer based on the data error collector, and the second data is displayed at the application layer.

[0104] In an optional manner, the method further includes: storing the running status information, the first data, and the second data persistently in a second storage layer of the database.

[0105] In an optional manner, if the running status information includes the running exception information, the first reporting step further includes, when the task is completed at a predetermined key node, obtaining stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder;

[0106] The step recorder is used to send the running status information to the application layer, the data error collector is used to send the first data to the application layer, and the running status information and the first data are displayed on the application layer, further comprising:

[0107] The running status information and the stack information are sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information, the stack information and the first data are displayed on the application layer.

[0108] In an optional manner, the key nodes include a first key node, a second key node, and a third key node with prompt priorities from high to low, the first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, the third key node includes a task configuration loading node, a Meta configuration loading node, a database table loading node, a data event merging node, and a task initialization node, and the display of the running status information on the application layer includes:

[0109] At the application layer, the abnormal operation information is prompted according to the prompt priority corresponding to the key node to which the abnormal operation information belongs.

[0110] In an alternative approach,

[0111] The first reporting step further includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtaining the running status information from the message queue;

[0112] The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

[0113] In an optional manner, the second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an identifier of no data error to the data error collector.

[0114] In an embodiment of the present invention, a task of cleaning data in a target system is performed based on a predetermined data cleaning tool; when the task is completed at a predetermined key node, the operation status information in the log cache is obtained, and the operation status information is reported to a preset step recorder, and the operation status information includes at least normal operation information and / or abnormal operation information; when the task is running, the data is detected based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if an erroneous first data is detected, the first data is reported to a preset data error collector; the operation status information is sent to the application layer based on the step recorder, and the first data is sent to the application layer based on the data error collector, and the operation status information and the first data are displayed on the application layer. This embodiment of the declaration can effectively track the link of data cleaning and quickly locate abnormal points, and check erroneous data.

[0115] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device, the computer device executes any of the above method embodiments.

[0116] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computer device to execute any of the above method embodiments.

[0117] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are run on a computer, the computer is caused to execute any of the above method embodiments.

[0118] The algorithm or demonstration provided herein are not inherently relevant to any particular computer, virtual system or other equipment. Various general-purpose systems may also be used together with the teachings based on this. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0119] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0120] Similarly, it should be understood that in order to streamline the present invention and facilitate understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0121] Those skilled in the art will appreciate that the modules in the computer device in the embodiment can be adaptively changed and set in one or more computer devices different from the embodiment. The modules or units or components in the embodiment can be combined into one module or unit or component, and can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or computer device disclosed so far can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0122] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A data processing link tracking method, characterized in that: The method comprises: Determine the target system and run a task of cleaning the data in the target system based on a predetermined data cleaning tool; When the task is running and after the task is finished, the first reporting step and the second reporting step are performed: The first reporting step includes, when the task is completed at a predetermined key node, obtaining operation status information in a log cache, and reporting the operation status information to a preset step recorder, wherein the operation status information includes at least normal operation information and / or abnormal operation information; The second reporting step includes, when the task is running, detecting the data based on a first detection rule predetermined in the data cleaning tool, and after the task is successfully completed, if erroneous first data is detected, reporting the first data to a preset data error collector; The running status information is sent to an application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information and the first data are displayed on the application layer.

2. The method according to claim 1, characterized in that The method further comprises: storing the cleaned data in a first storage layer of a database, and detecting the cleaned data in the first storage layer based on a second detection rule preset in the data error collector; If erroneous second data is detected, the second data is sent to the application layer based on the data error collector, and the second data is displayed at the application layer.

3. The method according to claim 2, characterized in that The method further includes: storing the operating status information, the first data, and the second data persistently in a second storage layer of the database.

4. The method according to claim 1, wherein If the running status information includes the running exception information, the first reporting step further includes, when the task is completed at a predetermined key node, obtaining stack information corresponding to the running exception information stored in the log cache, and reporting the running status information and the stack information to the step recorder; The step recorder is used to send the running status information to the application layer, the data error collector is used to send the first data to the application layer, and the running status information and the first data are displayed on the application layer, further comprising: The running status information and the stack information are sent to the application layer based on the step recorder, the first data is sent to the application layer based on the data error collector, and the running status information, the stack information and the first data are displayed on the application layer.

5. The method according to claim 4, characterized in that The key nodes include a first key node, a second key node, and a third key node with prompt priorities from high to low. The first key node includes a task running node, the second key node includes the first data acquisition node and the first data reporting node, and the third key node includes a task configuration loading node, a meta configuration loading node, a database table loading node, a data event merging node, and a task initialization node. Displaying the running status information on the application layer includes: At the application layer, the abnormal operation information is prompted according to the prompt priority corresponding to the key node to which the abnormal operation information belongs.

6. The method according to claim 1, characterized in that The first reporting step further includes, when the task is completed at a predetermined key node, obtaining the running status information stored in the log cache, creating an asynchronous first thread, pushing the running status information to a predetermined message queue based on the first thread, and the step recorder obtaining the running status information from the message queue; The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if an error in the first data is detected, creating an asynchronous second thread, pushing the first data to the message queue based on the second thread, and the data error collector obtains the first data from the message queue.

7. The method according to claim 1, characterized in that The second reporting step also includes, when the task is running, detecting the data based on the first detection rule in the data cleaning tool, and after the task is successfully completed, if no erroneous first data is detected, reporting an identifier of no data error to the data error collector.

8. A data processing link tracking device, characterized in that: include: An operation module is used to determine a target system and run a task of cleaning data in the target system based on a predetermined data cleaning tool; The reporting module is used to trigger the first reporting unit and the second reporting unit when the task is running and after the running is completed: The first reporting unit is configured to obtain operation status information in the log cache when the task is completed at a predetermined key node, and report the operation status information to a preset step recorder, wherein the operation status information includes at least normal operation information and / or abnormal operation information; The second reporting unit is configured to detect the data based on a first detection rule predetermined in the data cleaning tool, and report the first data to a preset data error collector if an erroneous first data is detected after the task is successfully completed; A display module is used to send the operating status information to the application layer based on the step recorder, send the first data to the application layer based on the data error collector, and display the operating status information and the first data on the application layer.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device, the computer device executes the method according to any one of claims 1 to 7.

Images