Data processing monitoring method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing monitoring method, electronic equipment, a storage medium and a program product. Relates to the big data field. The method comprises the steps that a first processing result generated by a scheduling system is obtained, the first processing result comprises a plurality of first data tables, and the first data tables are obtained by processing an original data set through the scheduling system; obtaining a plurality of historical processing results output by the calling system within a preset duration before the current moment, wherein the historical processing results comprise a plurality of historical data tables; according to the first processing result and the plurality of historical processing results, determining an operation state of the scheduling system; when it is determined that the operation state of the scheduling system is an abnormal state, metadata generated in the process of generating the first processing result by the scheduling system is obtained; and according to the metadata, the original data set and the plurality of first data tables, determining a data consanguinity map, and according to the data consanguinity map, determining an abnormality reason of the scheduling system abnormality. The method is used for achieving the effect of improving the monitoring efficiency.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a data processing monitoring method, electronic equipment, storage medium and program product. Background Art

[0002] In some scenarios, it is necessary to monitor the operating status of the system. For example, in a data scheduling scenario, it is necessary to monitor the scheduling system to determine whether the system is running normally.

[0003] In related technologies, staff can collect system logs, application logs, database logs and other log files through regular log review, and monitor the operating status of the system based on the log files.

[0004] However, since the above method relies on manual monitoring, system abnormalities cannot be discovered in a timely manner, resulting in low monitoring efficiency. Summary of the invention

[0005] The embodiments of the present application provide a data processing monitoring method, electronic device, storage medium and program product to achieve the effect of improving monitoring efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data processing monitoring method, comprising:

[0007] Obtaining a first processing result generated by the scheduling system, the first processing result including a plurality of first data tables, the first data tables being obtained by the scheduling system processing an original data set;

[0008] Acquire multiple historical processing results output by the calling system within a preset time period before the current moment, wherein the historical processing results include multiple historical data tables;

[0009] Determine an operating state of the scheduling system according to the first processing result and a plurality of historical processing results, wherein the operating state is a normal state or an abnormal state;

[0010] When determining that the operation state of the scheduling system is the abnormal state, obtaining metadata generated in the process of the scheduling system generating the first processing result;

[0011] A data lineage map is determined based on the metadata, the original data set and the plurality of first data tables, and an abnormal cause of the abnormality of the scheduling system is determined based on the data lineage map.

[0012] In a second aspect, an embodiment of the present application provides a data processing monitoring device, including: a first acquisition module, a second acquisition module, a first determination module, a third acquisition module and a second determination module, wherein:

[0013] The first acquisition module is used to acquire a first processing result generated by the scheduling system, the first processing result includes a plurality of first data tables, and the first data tables are obtained by the scheduling system processing the original data set;

[0014] The second acquisition module is used to acquire a plurality of historical processing results output by the calling system within a preset time period before the current moment, wherein the historical processing results include a plurality of historical data tables;

[0015] The first determination module is used to determine the operating state of the scheduling system according to the first processing result and multiple historical processing results, and the operating state is a normal state or an abnormal state;

[0016] The third acquisition module is used for acquiring metadata generated in the process of generating the first processing result by the scheduling system when determining that the operation state of the scheduling system is the abnormal state;

[0017] The second determination module is used to determine a data lineage map according to the metadata, the original data set and the plurality of first data tables, and to determine the abnormal cause of the abnormality of the scheduling system according to the data lineage map.

[0018] In a third aspect, an embodiment of the present application provides a data processing monitoring device, including: a memory, a processor;

[0019] The memory stores computer-executable instructions;

[0020] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0023] The embodiment of the present application provides a data processing monitoring method, electronic device, storage medium and program product. When it is necessary to monitor the operating status of the scheduling system, the operating status of the scheduling system can be determined based on the data table generated by the scheduling system and the data table generated historically. When the operating status is abnormal, the metadata generated in the process of generating the data table is obtained, and the abnormal cause of the abnormality of the scheduling system is determined through the data lineage map. The above data processing method can determine the abnormality of the scheduling system, and determine the abnormal cause based on the data lineage map, thereby improving the efficiency of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] Figure 1 A schematic diagram of the system architecture provided for an embodiment of the present application;

[0026] Figure 2 A flowchart of a data processing monitoring method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a visual operation window of a scheduling system provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a process for determining the operating status of a scheduling system provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a process for determining the cause of an abnormality in a scheduling system provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a data processing monitoring device provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0036] It should be noted that the data processing monitoring method, electronic device, storage medium and program product provided in the present application can be used in the field of big data, and can also be used in any field except the field of big data. The application field of the data processing monitoring method, electronic device, storage medium and program product in the present application is not limited.

[0037] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0038] For ease of understanding, the following Figure 1 , the system architecture applicable to the embodiments of the present application is described.

[0039] Figure 1 This is a schematic diagram of the system architecture provided by the embodiment of this application. Figure 1, including a data collection module 101, a monitoring module 102, a lineage analysis module 103 and a notification module 104. The data collection module 101 can be used to collect data generated by the scheduling system, the monitoring module 102 can be used to monitor the operating status of the scheduling system, the lineage analysis module 103 can be used to determine the data lineage map and determine the cause of the abnormality of the scheduling system when an abnormality of the scheduling system is detected, and the notification module 104 can send an abnormality notification to the user device after an abnormality of the scheduling system is detected.

[0040] In related technologies, staff can collect system logs, application logs, database logs and other log files through regular log review, and monitor the operating status of the system based on the log files. However, since the above method relies on manual monitoring, it is not possible to detect system anomalies in a timely manner, resulting in low monitoring efficiency.

[0041] In view of the above technical problems, in an embodiment of the present application, when it is necessary to monitor the operating status of the scheduling system, the operating status of the scheduling system can be determined based on the data table generated by the scheduling system and the data table generated historically, and when the operating status is abnormal, the metadata generated in the process of generating the data table is obtained, and the abnormal cause of the abnormality of the scheduling system is determined through the data lineage map. The above data processing method can determine the abnormality of the scheduling system, and determine the abnormal cause according to the data lineage map, thereby improving the efficiency of monitoring.

[0042] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0043] Figure 2 A flow chart of a data processing monitoring method provided in an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0044] S201. Obtain a first processing result generated by a scheduling system.

[0045] The execution subject of the embodiment of the present application can be a terminal device, or a data processing monitoring device set in the terminal device. The data processing monitoring device can be implemented by software, or by a combination of software and hardware.

[0046] The terminal device may be provided with a dispatching system, which may be used to manage data and data processing tasks. For example, the dispatching system may perform data extraction, data conversion, data addition, data analysis, report generation, etc. The visual operation interface of the dispatching system may have a "generate" control, and the user may generate a data table by clicking the "generate" control.

[0047] Next, combine Figure 3 , through specific examples, the visual operation window of the scheduling system is explained.

[0048] Figure 3 For a schematic diagram of a visual operation window of a scheduling system provided in an embodiment of the present application, see Figure 3 , including a data set selection module and a "generate" control. Users can select the original data set that needs to be processed in the data set selection module. By clicking the "generate" control, the scheduling system can process the original data set and generate multiple data tables.

[0049] The first processing result may include multiple first data tables, which may be obtained by the scheduling system processing the original data set, and the format of the first data table may be a relational database table, a flat file, a large data format table, etc. The first data table may be stored in the storage space of the terminal device.

[0050] Multiple first data tables can be determined in the following manner: obtain an original data set, which may refer to a set of original data, such as a product data set, a smart device data set, etc.; perform data cleaning on the original data set to obtain a cleaned data set, and the data cleaning process may include removing duplicate data, filling missing values, correcting data format errors, etc.; perform data conversion on the cleaned data set, and the data conversion process may include converting the data format in the cleaned data set into a preset format of the data table to obtain a converted data set; perform data integration on the converted data set, and the data integration process may include integrating the data in the converted data set to obtain multiple first data tables.

[0051] S202: Acquire multiple historical processing results output by the calling system within a preset time period before the current moment.

[0052] The historical processing results may include multiple historical data tables, which may be obtained from the storage space of the terminal device. The historical data table may refer to a call to a system generated at a historical moment and obtained by processing a historical data set.

[0053] The preset duration may refer to a value set by the user in the terminal device in advance, for example, the preset duration may be 3 days. For example, assuming that the current time is October 13, 2024, multiple historical processing results output from October 10, 2024 to October 13, 2024 are obtained.

[0054] S203: Determine the operating status of the scheduling system according to the first processing result and multiple historical processing results.

[0055] The historical processing result operation status may include a normal state or an abnormal state.

[0056] The operating status of the scheduling system can be determined in the following manner: determine multiple data table types based on a first processing result and multiple historical processing results; for any data table type, determine multiple data tables corresponding to the data table type in the first processing result and multiple historical processing results; determine a verification result corresponding to the data table type based on the multiple data tables corresponding to the data table type, the verification result is used to indicate whether there are abnormalities in the multiple data tables corresponding to the data table type; determine the operating status based on the verification result corresponding to each data table type.

[0057] The data table type may refer to a type corresponding to the data table, for example, the data table type may include a transaction type, a business type, an order type, etc. The verification result may indicate whether the multiple data tables are abnormal or not. The operation status may include a normal status and an abnormal status.

[0058] S204: When it is determined that the operation state of the scheduling system is an abnormal state, metadata generated in the process of the scheduling system generating the first processing result is obtained.

[0059] Metadata can refer to data that describes the original data. Metadata can include job time, job name, query statement, user information, report information, etc.

[0060] The metadata can be determined in the following ways: obtaining the original data set; determining the metadata type, and according to the metadata type, determining the operation time, operation name, query statement, user information, report information and other data corresponding to the original data set; determining the operation time, operation name, query statement, user information, report information and other data corresponding to the original data set as metadata. The metadata can be represented in a table.

[0061] For example, assuming that the metadata type is scheduling metadata, the metadata type may include scheduling job, job name, scheduling date, execution status, and dependency, as shown in Table 1. See Table 1.

[0062] Table 1

[0063]

[0064] S205. Determine a data lineage map based on the metadata, the original data set, and the plurality of first data tables, and determine the cause of the abnormality of the scheduling system based on the data lineage map.

[0065] The data lineage map can refer to a visual representation of the path of flow and transformation of the original data in the original data set within the scheduling system. The data lineage map can describe the flow of the original data from the source to the final destination. The data lineage map can include each stage of the generation, transmission, conversion and storage of the original data.

[0066] The data lineage map can be determined in the following way: determine the data type of each data in the original data set to obtain multiple data types; determine the first relationship between the multiple data types, the second relationship between the multiple data types and the multiple first data tables, and the third relationship between the multiple first data tables based on the metadata; construct a data lineage map based on the first relationship, the second relationship, and the third relationship, the data lineage map including nodes corresponding to each data type, nodes corresponding to each first data table, and connection relationships between the nodes.

[0067] The data type may be a user data type, a business data type, an order data type, etc. The first relationship may be used to indicate the dependency relationship between multiple data types in metadata; the second relationship is used to indicate the mapping relationship between data types in a data table; and the third relationship is used to indicate the data flow relationship between different data tables.

[0068] In an embodiment of the present application, when it is necessary to monitor the operating status of the scheduling system, a first processing result generated by the scheduling system can be obtained, wherein the first processing result includes multiple first data tables; multiple historical processing results output by the calling system within a preset time period before the current moment are obtained, wherein the historical processing results include multiple historical data tables; the operating status of the scheduling system is determined based on the first processing result and the multi-core historical processing results, and the operating status of the scheduling system includes a normal state or an abnormal state; if the operating status of the scheduling system is an abnormal state, metadata generated in the process of the scheduling system generating the first processing result is obtained; based on the metadata, the original data set and multiple first data tables, a data lineage map is determined, and based on the data lineage map, the cause of the abnormality of the scheduling system is determined. The above method can determine the abnormality of the scheduling system by comparing the first data table and the historical data table, and determine the cause of the abnormality based on the data lineage map, thereby improving the efficiency of monitoring the scheduling system.

[0069] Based on any of the above embodiments, Figure 4 , the process of determining the operating status of the dispatching system ( Figure 2 S203) in the embodiment is described in detail.

[0070] Figure 4 Schematic diagram of the process of determining the operating status of the scheduling system provided in the embodiment of the present application. Figure 4 , the method may include:

[0071] S401. Determine multiple data table types according to a first processing result and multiple historical processing results.

[0072] The first processing result may include multiple first data tables; the historical processing result may include multiple historical data tables. The data table type may refer to the type corresponding to the data table, for example, the data table type may include transaction type, business type, order type, etc.

[0073] S402: For any data table type, determine multiple data tables corresponding to the data table type from the first processing result and multiple historical processing results.

[0074] The multiple data tables can be determined in the following manner: obtain multiple first data tables corresponding to the first processing result, and search for a first target data table of the corresponding data table type in the multiple first data tables; obtain multiple historical data tables corresponding to multiple historical processing results, and search for multiple historical target data tables of the corresponding data table type in the multiple historical data tables, and determine the first target data table and the multiple historical target data tables as the multiple data tables corresponding to the data table type.

[0075] For example, assuming that the data table type is a business type, then a first target data table whose data table type is a business type is searched among multiple first data tables, and multiple historical target data tables whose data table type is a business type are searched among multiple historical data tables, and the first target data table and multiple historical target data tables are determined as multiple data tables corresponding to the business type.

[0076] S403: Determine multiple fields in each data table, and determine whether table structures of multiple data tables are consistent based on the multiple fields in each data table.

[0077] The table structure can define the composition and basic framework of the data table. The table structure can include field names, data types, constraints, etc. Constraints can include primary keys, foreign keys, uniqueness, non-null, default values, etc.

[0078] Whether the table structures of multiple data tables are consistent can be determined in the following ways: using a field extraction tool to obtain multiple fields in each data table; matching the field names of multiple data tables to obtain a first result; matching the data types of multiple data tables to obtain a second result; matching the constraints of multiple data tables to obtain a third result; if the first result, the second result, and the third result all indicate a match, it is determined that the table structures of the multiple data tables are consistent. The first result, the second result, and the third result can indicate that the multiple data tables match or do not match.

[0079] If yes, execute S404.

[0080] If not, execute S405.

[0081] S404: Obtain similarities of multiple data tables, and determine whether the similarities are greater than or equal to a preset threshold.

[0082] The similarity between data tables can be determined based on whether the contents of the data tables are similar. The value range of the similarity can be 0 to 1. The greater the similarity, the more similar the contents between the data tables are.

[0083] The similarity of multiple data tables can be determined in the following manner: for the data distribution of the fields in any two data tables, determine the target similarity between every two data tables by statistical methods according to the data distribution, where the statistical methods may include histograms, quantiles, etc.; obtain multiple target similarities between every two data tables in turn, calculate the average of the multiple target similarities, and obtain the similarity of the multiple data tables.

[0084] The preset threshold value may refer to a value set in advance by the user in the terminal device. For example, the preset threshold value may be 0.8.

[0085] If yes, execute S405

[0086] If not, execute S406

[0087] S405: Determine whether the verification result indicates that multiple data tables are abnormal.

[0088] S406: Determine whether the verification result indicates that there is no abnormality in the plurality of data tables.

[0089] S407: Determine the running status according to the verification result corresponding to each data table type.

[0090] The operating status may include an abnormal status and a normal status.

[0091] The running status can be determined in the following manner: if among multiple data table types, the verification results corresponding to the data table types indicate that the corresponding multiple data tables have abnormalities, then the running status is determined to be an abnormal state; if among multiple data table types, the verification results corresponding to each data table type respectively indicate that the corresponding multiple data tables do not have abnormalities, then the running status is determined to be a normal state.

[0092] In an embodiment of the present application, when it is necessary to monitor the operating status of the scheduling system, the first processing result and multiple historical processing results of the scheduling system can be obtained first, and multiple data table types can be determined based on the first processing result and the multiple historical processing results; for any data table type, multiple data tables corresponding to the data table type are determined in the first processing result and the multiple historical processing results; multiple fields existing in each data table are determined, and based on the multiple fields existing in each data table, it is determined whether the table structures of the multiple data tables are consistent; if not, it indicates that the table structure has changed, and the verification result indicates that there are abnormalities in the multiple data tables; if so, it is obtained Take the similarity of multiple data tables, and determine whether the similarity is greater than or equal to a preset threshold. If the similarity is greater than or equal to the preset threshold, it indicates that the data table has not been updated within the preset time, and the verification result indicates that multiple data tables have abnormalities; if the similarity is less than the preset threshold, it is determined that the verification result indicates that multiple data tables do not have abnormalities; if the verification result corresponding to the data table type in multiple data table types indicates that the corresponding multiple data tables have abnormalities, it is determined that the operating state is an abnormal state; if the verification results corresponding to each data table type in multiple data table types indicate that the corresponding multiple data tables do not have abnormalities, it is determined that the operating state is a normal state. Through the above method, the operating state of the dispatching system can be determined by judging the table structure and similarity of multiple data tables, thereby improving the efficiency of dispatching system monitoring.

[0093] Based on any of the above embodiments, Figure 5 , the process of determining the cause of abnormality in the dispatching system ( Figure 2 S205) in the embodiment is described in detail.

[0094] Figure 5 Schematic diagram of the process of determining the cause of the abnormality of the scheduling system provided in the embodiment of the present application. Figure 5 , the method may include:

[0095] S501. Determine the data type of each data in the original data set according to the original data set to obtain multiple data types.

[0096] It should be noted that the execution process of the above S501 can refer to S205, which will not be repeated here.

[0097] S502: Determine, according to the metadata, a first relationship between multiple data types, a second relationship between multiple data types and multiple first data tables, and a third relationship between the multiple first data tables.

[0098] The first relationship can be used to indicate the dependency relationship between multiple data types in metadata; the second relationship is used to indicate the mapping relationship between data types in the data table; and the third relationship is used to indicate the data flow relationship between different data tables.

[0099] The first relationship can be determined in the following manner: obtaining metadata; determining the derived relationship between data types according to the metadata; and determining the derived relationship as the first relationship. For example, assuming that the first data type is a string and the second data type is a date, the first data type can be converted to the second data type, and the first relationship is a conversion relationship.

[0100] The second relationship may be determined in the following manner: determining a query statement in the metadata according to the metadata; determining an interactive relationship between the data type and the data table according to the query statement; and determining the interactive relationship as the second relationship. For example, the third data type may be loaded into the first data table through extraction processing.

[0101] The third relationship may be determined in the following manner: according to the metadata, a data flow relationship between the data tables is determined, and the data flow relationship is determined as the third relationship. For example, the data flow relationship may be that the first data table is provided as input to the other data tables, or the second data table depends on the data of the third data table.

[0102] S503: Construct a data lineage map based on the first relationship, the second relationship and the third relationship.

[0103] The data lineage graph includes nodes corresponding to each data type, nodes corresponding to each first data table, and connection relationships between nodes.

[0104] The data lineage map can be constructed in the following way: create a node for each data type and each first data table; construct a connection relationship between each node based on the first relationship, the second relationship and the third relationship to obtain an initial data lineage map, where the connection relationship may refer to the connecting lines and edges between the nodes.

[0105] The constructed data lineage map can be stored in the storage space of the terminal device.

[0106] S504: Determine an abnormality type of the scheduling system, and determine an abnormal node in data types corresponding to the plurality of first data tables and / or the original data set according to the abnormality type.

[0107] Exception types may include table structure change type or data not updated type.

[0108] The exception type can be determined in the following way: obtain the table structure of multiple data tables. If the table structures of the multiple data tables are inconsistent, it indicates that the exception type of the scheduling system is the table structure change type; obtain the similarity of multiple data tables. If the similarity is greater than or equal to a preset threshold, it indicates that the exception type of the scheduling system is the data not updated type.

[0109] The abnormal node is any first data table in the plurality of first data tables, or any data type in the original data set.

[0110] The abnormal node can be determined in the following manner: if the abnormal type is a table structure change type, a target data table is determined from multiple first data tables, at least one data type corresponding to the target data table is determined, and the target data table and at least one data type are determined as abnormal nodes, and the table structure of the target data table has changed; if the abnormal type is a data not updated type, at least one target data type is determined from the data types corresponding to the original data set, and at least one target data type is determined as an abnormal node, and the data of the target data type has not been updated.

[0111] S505: Determine the associated node corresponding to the abnormal node in the data lineage graph.

[0112] The associated nodes may include directly associated nodes having a direct connection relationship with the abnormal node and indirectly associated nodes indirectly connected to the abnormal node through a plurality of nodes.

[0113] The associated nodes may be determined in the following manner: obtaining the abnormal node and determining the position of the abnormal node in the data lineage map; determining the directly associated nodes that have a direct connection relationship with the abnormal node and determining the indirectly associated nodes that are indirectly connected to the abnormal node through multiple nodes based on the position of the abnormal node in the data lineage map; determining the directly associated nodes and the indirectly associated nodes as the associated nodes corresponding to the abnormal node.

[0114] S506: Determine the cause of the abnormality according to the abnormal node and the associated nodes.

[0115] The cause of the exception can be determined in the following ways: based on the abnormal node and the associated nodes, obtain the historical log of the scheduling system; based on the historical log, determine the change history of the abnormal node and the associated nodes, for example, the change history may include configuration changes or code changes, etc.; based on the change history, determine the cause of the exception, for example, the cause of the exception may include configuration errors, code defects, system failures, etc.

[0116] Optionally, after determining the cause of the exception, the cause of the exception, the abnormal node and the associated nodes can be input into a large language model to obtain a recommended solution output by the large language model; according to the metadata, the user information corresponding to the abnormal node and the associated node is searched, and according to the user information, the target user and the user device corresponding to the target user are determined; the abnormal node, the associated node, the cause of the exception and the recommended solution are sent to the user device in the form of a message.

[0117] In an embodiment of the present application, when it is necessary to monitor the operating status of the scheduling system, the cause of the abnormality of the scheduling system can be determined, and based on the original data set, the data type of each data in the original data set can be determined to obtain multiple data types; based on the metadata, the first relationship between multiple data types, the second relationship between multiple data types and multiple first data tables, and the third relationship between multiple first data tables can be determined; based on the first relationship, the second relationship, and the third relationship, a data lineage map can be constructed; the abnormal type of the scheduling system is determined, and based on the abnormal type, the abnormal node can be determined in the data type corresponding to the multiple first data tables and / or the original data set; based on the abnormal node, the associated node corresponding to the abnormal node can be determined in the data lineage map; based on the abnormal node and the associated node, the cause of the abnormality can be determined, and a recommended solution can also be determined based on the abnormal cause, and the abnormal node, the associated node, the cause of the abnormality and the recommended solution can be sent to the user device in the form of a message; through the above method, the abnormal node and the associated node can be determined according to the data lineage map, and then the cause of the abnormality of the scheduling system can be determined, thereby improving the efficiency of monitoring the scheduling system.

[0118] Figure 6 A schematic diagram of a data processing monitoring device provided in an embodiment of the present application. Figure 6 The data processing monitoring device 10 includes: a first acquisition module 11, a second acquisition module 12, a first determination module 13, a third acquisition module 14 and a second determination module 15, wherein:

[0119] The first acquisition module 11 is used to acquire a first processing result generated by the scheduling system, wherein the first processing result includes a plurality of first data tables, and the first data tables are obtained by the scheduling system processing the original data set;

[0120] The second acquisition module 12 is used to acquire a plurality of historical processing results output by the calling system within a preset time period before the current moment, wherein the historical processing results include a plurality of historical data tables;

[0121] The first determination module 13 is used to determine the operating state of the scheduling system according to the first processing result and multiple historical processing results, and the operating state is a normal state or an abnormal state;

[0122] The third acquisition module 14 is used to acquire metadata generated in the process of generating the first processing result by the scheduling system when it is determined that the operation state of the scheduling system is the abnormal state;

[0123] The second determination module 15 is used to determine a data lineage map according to the metadata, the original data set and the plurality of first data tables, and determine the abnormal cause of the abnormality of the scheduling system according to the data lineage map.

[0124] A data processing monitoring device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0125] In a possible design, the first determining module 13 is specifically configured to:

[0126] Determining multiple data table types according to the first processing result and the multiple historical processing results;

[0127] For any data table type, determining multiple data tables corresponding to the data table type from the first processing result and the multiple historical processing results;

[0128] Determine, according to the multiple data tables corresponding to the data table type, a verification result corresponding to the data table type, wherein the verification result is used to indicate whether there are abnormalities in the multiple data tables corresponding to the data table type;

[0129] The operating status is determined according to the verification result corresponding to each data table type.

[0130] In a possible design, the first determining module 13 is specifically configured to:

[0131] Determine the multiple fields that exist in each data table;

[0132] According to multiple fields in each data table, determining whether the table structures of the multiple data tables are consistent;

[0133] If yes, obtain the similarity of the multiple data tables, and if the similarity of the multiple data tables is greater than or equal to a preset threshold, determine that the verification result indicates that the multiple data tables have anomalies; if the similarity of the multiple data tables is less than the preset threshold, determine that the verification result indicates that the multiple data tables do not have anomalies;

[0134] If not, it is determined that the verification result indicates that anomalies exist in the plurality of data tables.

[0135] In a possible design, the first determining module 13 is specifically configured to:

[0136] If, among the multiple data table types, a plurality of data tables corresponding to the verification result indication corresponding to the data table type exist abnormalities, then determining that the operating state is the abnormal state;

[0137] If the verification results corresponding to each data table type in the multiple data table types respectively indicate that there is no abnormality in the corresponding multiple data tables, it is determined that the operating state is the normal state.

[0138] In a possible design, the second determining module 15 is specifically configured to:

[0139] Determine the abnormal type of the scheduling system, the abnormal type table structure change type or data not updated type;

[0140] According to the abnormal type, determining an abnormal node in the data types corresponding to the multiple first data tables and / or the original data set, the abnormal node being any first data table in the multiple first data tables or any data type in the original data set;

[0141] Determining the associated node corresponding to the abnormal node in the data lineage graph;

[0142] The abnormal cause is determined according to the abnormal node and the associated node.

[0143] In a possible design, the second determining module 15 is specifically configured to:

[0144] If the abnormal type is the table structure change type, determining a target data table from among the multiple first data tables, determining at least one data type corresponding to the target data table, and determining the target data table and the at least one data type as the abnormal nodes, and the table structure of the target data table changes;

[0145] If the exception type is the data not updated type, at least one target data type is determined in the data types corresponding to the original data set, and the at least one target data type is determined as the exception node, and the data of the target data type is not updated.

[0146] A data processing monitoring device provided in an embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0147] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 20 may include: a transceiver 21 , a processor 22 , and a memory 23 .

[0148] The processor 22 executes the computer execution instructions stored in the memory, so that the processor 22 executes the scheme in the above embodiment. The processor 22 can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0149] The memory 23 is connected to the processor 22 via a system bus and completes communication between them. The memory 23 is used to store computer program instructions.

[0150] The transceiver 21 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0151] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0152] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0153] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solution of the data processing monitoring method in the above embodiment.

[0154] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the data processing monitoring method in the above embodiment.

[0155] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the data processing monitoring method in the above embodiment.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0157] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0158] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0159] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0160] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0161] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0162] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0163] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0164] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0165] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing monitoring method, characterized in that: include: Obtaining a first processing result generated by the scheduling system, the first processing result including a plurality of first data tables, the first data tables being obtained by the scheduling system processing an original data set; Acquire multiple historical processing results output by the calling system within a preset time period before the current moment, wherein the historical processing results include multiple historical data tables; Determine an operating state of the scheduling system according to the first processing result and a plurality of historical processing results, wherein the operating state is a normal state or an abnormal state; When determining that the operation state of the scheduling system is the abnormal state, obtaining metadata generated in the process of the scheduling system generating the first processing result; A data lineage map is determined based on the metadata, the original data set and the plurality of first data tables, and an abnormal cause of the abnormality of the scheduling system is determined based on the data lineage map.

2. The method according to claim 1, characterized in that Determining the operating state of the scheduling system according to the first processing result and a plurality of historical processing results includes: Determining multiple data table types according to the first processing result and the multiple historical processing results; For any data table type, determining multiple data tables corresponding to the data table type from the first processing result and the multiple historical processing results; Determine, according to the multiple data tables corresponding to the data table type, a verification result corresponding to the data table type, wherein the verification result is used to indicate whether there are abnormalities in the multiple data tables corresponding to the data table type; The operating status is determined according to the verification result corresponding to each data table type.

3. The method according to claim 2, characterized in that Determining, according to a plurality of data tables corresponding to the data table type, a verification result corresponding to the data table type, including: Determine the multiple fields that exist in each data table; According to multiple fields in each data table, determining whether the table structures of the multiple data tables are consistent; If yes, obtain the similarity of the multiple data tables, and if the similarity of the multiple data tables is greater than or equal to a preset threshold, determine that the verification result indicates that the multiple data tables have anomalies; if the similarity of the multiple data tables is less than the preset threshold, determine that the verification result indicates that the multiple data tables do not have anomalies; If not, it is determined that the verification result indicates that anomalies exist in the plurality of data tables.

4. The method according to claim 2 or 3, characterized in that: Determining the operating state according to the verification result corresponding to each data table type includes: If, among the multiple data table types, a plurality of data tables corresponding to the verification result indication corresponding to the data table type exist abnormalities, then determining that the operating state is the abnormal state; If the verification results corresponding to each data table type in the multiple data table types respectively indicate that there is no abnormality in the corresponding multiple data tables, it is determined that the operating state is the normal state.

5. The method according to any one of claims 1 to 4, characterized in that: Determining a data lineage map according to the metadata, the original data set, and the plurality of first data tables includes: Determine the data type of each data in the original data set to obtain multiple data types; Determining, according to the metadata, a first relationship between the plurality of data types, a second relationship between the plurality of data types and the plurality of first data tables, and a third relationship between the plurality of first data tables; The data lineage graph is constructed based on the first relationship, the second relationship and the third relationship. The data lineage graph includes nodes corresponding to each data type, nodes corresponding to each first data table, and connection relationships between nodes.

6. The method according to any one of claims 1 to 5, characterized in that: Determining the abnormal cause of the abnormality of the scheduling system according to the data lineage map includes: Determine the abnormal type of the scheduling system, the abnormal type table structure change type or data not updated type; According to the abnormal type, determining an abnormal node in the data types corresponding to the multiple first data tables and / or the original data set, the abnormal node being any first data table in the multiple first data tables or any data type in the original data set; Determining the associated node corresponding to the abnormal node in the data lineage graph; The abnormal cause is determined according to the abnormal node and the associated node.

7. The method according to claim 6, characterized in that Determining an abnormal node in the data types corresponding to the plurality of first data tables and / or the original data set according to the abnormal type includes: If the abnormal type is the table structure change type, determining a target data table from among the multiple first data tables, determining at least one data type corresponding to the target data table, and determining the target data table and the at least one data type as the abnormal nodes, and the table structure of the target data table changes; If the exception type is the data not updated type, at least one target data type is determined in the data types corresponding to the original data set, and the at least one target data type is determined as the exception node, and the data of the target data type is not updated.

8. A data processing monitoring device, characterized in that: include: A first acquisition module, a second acquisition module, a first determination module, a third acquisition module and a second determination module, wherein: The first acquisition module is used to acquire a first processing result generated by the scheduling system, the first processing result includes a plurality of first data tables, and the first data tables are obtained by the scheduling system processing the original data set; The second acquisition module is used to acquire a plurality of historical processing results output by the calling system within a preset time period before the current moment, wherein the historical processing results include a plurality of historical data tables; The first determination module is used to determine the operating state of the scheduling system according to the first processing result and multiple historical processing results, and the operating state is a normal state or an abnormal state; The third acquisition module is used for acquiring metadata generated in the process of generating the first processing result by the scheduling system when determining that the operation state of the scheduling system is the abnormal state; The second determination module is used to determine a data lineage map according to the metadata, the original data set and the plurality of first data tables, and to determine the abnormal cause of the abnormality of the scheduling system according to the data lineage map.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images