Data monitoring method and device

Abstract

The invention discloses a data monitoring method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the following steps: receiving operator index data, wherein the operator index data comprises a first operator identifier and first index data corresponding to the first operator identifier; according to first index data corresponding to the first operator identifier, first edge data with the first operator identifier as a vertex in the graph database is updated; in response to monitoring of triggering of the timed task, the timed task indicates a second operator identifier, and second edge data corresponding to the second operator identifier is acquired from the graph database; and when the second edge data does not meet the preset condition corresponding to the second operator identifier, generating early warning information corresponding to the second operator identifier. According to the embodiment, the operator index data is imported into the graph database, comparison of the preset conditions is carried out in real time, the early warning effect is achieved, active monitoring is achieved, meanwhile, rapid positioning of the data quality problem is achieved, and the data quality troubleshooting efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data monitoring method and device. Background Art

[0002] Business or usage scenarios based on data platforms such as logistics risk control platforms require data platforms to process large amounts of data to provide support for various businesses. Since multiple technologies or tools are used in the data processing process, the data quality cannot be guaranteed, so data quality needs to be monitored.

[0003] In the related technology, for real-time scenarios, when problems occur, the downstream system will feedback the problems to the upstream system step by step. There is a lack of corresponding early warning mechanism and effective data quality problem location mechanism, and the error location cannot be located quickly. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data monitoring method and device, which imports operator indicator data into a graph database, performs real-time comparison of preset conditions, achieves an early warning effect, realizes active monitoring, and simultaneously realizes rapid location of data quality problems, thereby improving the efficiency of data quality troubleshooting.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data monitoring method is provided, comprising:

[0006] Receive operator indicator data, where the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier;

[0007] Update first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier;

[0008] In response to monitoring the triggering of a scheduled task, the scheduled task indicates a second operator identifier, acquiring second edge data corresponding to the second operator identifier from the graph database;

[0009] When the second edge data does not satisfy a preset condition corresponding to the second operator identifier, a warning message corresponding to the second operator identifier is generated.

[0010] Optionally, before receiving operator indicator data, the following steps are included:

[0011] An operator indicator template is defined, wherein the operator indicator template includes a field indicating an operator identifier, a field indicating indicator data, and a field indicating external service information, wherein the operator indicator data also includes external service information corresponding to the operator identifier.

[0012] Optionally, receiving the operator indicator data includes: receiving the operator indicator data regularly reported by a reporter;

[0013] Before receiving operator indicator data, it also includes:

[0014] A new graph database reporting class is added, in which the graph database reporting class indicates the method by which the reporter reports the operator indicator data to the graph database.

[0015] Optionally, before updating the first edge data with the first operator identifier as a vertex in the graph database according to the first indicator data corresponding to the first operator identifier, the method includes:

[0016] receiving third operator indicator data, wherein the third operator indicator data includes a first operator identifier and third indicator data corresponding to the first operator identifier;

[0017] Acquire a data flow relationship corresponding to the first operator identifier;

[0018] A graph structure corresponding to the first operator identifier is constructed according to the first operator identifier, the third indicator data, and the data flow relationship.

[0019] Optionally, constructing a graph structure corresponding to the first operator identifier according to the first operator identifier, the third indicator data, and the data flow relationship includes:

[0020] Acquire a third operator identifier downstream of the first operator identifier from the data flow relationship;

[0021] The graph structure is constructed by taking the first operator identifier and the third operator identifier as vertices respectively and taking the third indicator data as edge data pointing from the first operator identifier to the third operator identifier.

[0022] Optionally, updating first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier includes:

[0023] A graph structure including vertices corresponding to the first operator identifier is obtained, first edge data corresponding to the first operator identifier is obtained from the graph structure, and the first edge data is updated according to the first indicator data.

[0024] Optionally, generating warning information corresponding to the second operator identifier includes:

[0025] The warning information is generated according to the second operator identifier, the second edge data, and the external service information.

[0026] According to another aspect of an embodiment of the present invention, there is provided a data monitoring device, comprising:

[0027] A receiving module receives operator indicator data, where the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier;

[0028] An updating module, which updates first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier;

[0029] an acquisition module, in response to monitoring the triggering of a scheduled task, wherein the scheduled task indicates a second operator identifier, acquiring second edge data corresponding to the second operator identifier from the graph database;

[0030] A generating module is configured to generate warning information corresponding to the second operator identifier when the second edge data does not satisfy a preset condition corresponding to the second operator identifier.

[0031] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data monitoring method provided by the present invention.

[0032] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data monitoring method provided by the present invention is implemented.

[0033] One embodiment of the above invention has the following advantages or beneficial effects: the data monitoring method provided by the embodiment of the present invention, after receiving the operator indicator data, updates the first edge data with the first operator identifier as the vertex in the graph database according to the first indicator data corresponding to the first operator identifier of the operator indicator data. After monitoring the departure of the scheduled task, the second edge data corresponding to the second operator identifier indicated by the scheduled task is obtained from the graph database, and when the second edge data does not meet the preset conditions corresponding to the second operator identifier, the warning information corresponding to the second operator identifier is generated. This method achieves the effect of early warning for data quality monitoring by writing the operator indicator data into the graph database and performing real-time judgment of the preset conditions. The graph database can quickly locate data quality problems; this method realizes flexible monitoring of data quality problems by combining the stream processing framework with the graph database, can realize active monitoring of data quality, and improves the efficiency of data quality troubleshooting.

[0034] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0036] Figure 1 is a schematic diagram of the main process of a data monitoring method according to an embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of the main process of another data monitoring method according to an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of the main process of a data monitoring method according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of main modules of a data monitoring device according to an embodiment of the present invention;

[0040] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] Data platforms such as logistics risk control platforms support many models for scenarios such as transportation capacity and merchants. The clues produced by each model are synchronized to different business systems in the form of messages. Risk control clues can be divided into offline and real-time according to the output cycle. Offline technology and corresponding data quality methods are relatively mature, while the data quality of real-time risk control scenarios lacks a unified standard solution. When problems occur in the output of real-time risk control clues, the downstream system will feedback and troubleshoot the problems step by step. There is no early warning mechanism and the efficiency is low. In addition, the stream data framework processing technology used by the logistics risk control platform uses external calls to make dynamic judgments on clues, making it more difficult to quickly locate data quality problems. That is, in related technologies, for business indicators such as external calls or cache queries, data quality monitoring cannot be performed, and there is a lack of effective data quality problem location mechanism, making it difficult to quickly locate the error location, and there is a lack of corresponding early warning mechanism when problems occur. For dynamic judgment logic combined with logs to determine problem points, the method of locating problems is relatively cumbersome and inefficient.

[0044] Based on the above problems, an embodiment of the present invention provides a data monitoring method, which can monitor the data quality of dynamic judgment logic, quickly locate problems, and issue early warnings when problems occur, with high efficiency.

[0045] Figure 1 is a schematic diagram of the main process of a data monitoring method according to an embodiment of the present invention, such as Figure 1 As shown, the data monitoring method comprises the following steps:

[0046] Step S101: receiving operator indicator data, where the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier;

[0047] Step S102: updating the first edge data with the first operator identifier as a vertex in the graph database according to the first indicator data corresponding to the first operator identifier;

[0048] Step S103: in response to monitoring the triggering of the scheduled task, the scheduled task indicates a second operator identifier, and obtaining second edge data corresponding to the second operator identifier from the graph database;

[0049] Step S104: when the second edge data does not satisfy the preset condition corresponding to the second operator identifier, generate warning information corresponding to the second operator identifier.

[0050] In an embodiment of the present invention, the data monitoring method can monitor the data quality of risk control clues in real-time scenarios, wherein the risk control clues are various information or indicators used to identify and evaluate potential risks.

[0051] In an embodiment of the present invention, in the data monitoring method, operator indicator data is first received, and the operator indicator data may come from a stream processing framework, that is, the operator indicator data may be sent or synchronized by the stream processing framework, and the stream processing framework may be Flink (an open source, distributed, high-performance, fault-tolerant stream processing framework), which has low-latency event-driven real-time computing capabilities. In the stream processing framework, an operator is a computing process that converts one or more data streams into a new data stream, and the combination of multiple operators can realize the conversion logic of complex data streams.

[0052] The operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier. The first operator identifier is the identifier of the operator that performs the calculation. The first operator identifier includes a first operator name and a first operator code. The first operator name may include a first operator type. The first indicator data includes a first indicator and an indicator result corresponding to the first indicator. The first indicator can be set according to the operator type. Among them, Flink operator types include source, filter, map, process, window, and sink. The first indicator can include the operator input data volume and the operator output data volume. If the operator type is a filter type (filter), the first indicator can also include the risk control clue hit data volume and the risk control clue miss data volume. Operator indicator data may also include external service information, which includes the type of external service that the current calculation depends on, and may also include detailed information corresponding to the external service type, where the external service type may include JSF (Java build framework), Redis (key-value database), Groovy (an agile dynamic language for Java virtual machine), etc. The detailed information corresponding to the external service type is determined according to the external service type. For example, when the external service type is JSF, it means that the current calculation depends on JSF, and the corresponding detailed information is the interface name; when the external service type is Redis, the corresponding detailed information may be the stored information category, such as basic address information, basic employee information, etc.; when the external service type is Groovy, the detailed information may be the Groovy script name. Operator indicator data may also include the reporting time of the operator indicator data.

[0053] In an embodiment of the present invention, before receiving operator indicator data, it includes: defining an operator indicator template, the operator indicator template includes a field indicating an operator identifier, a field indicating indicator data, and a field indicating external service information, and the operator indicator data also includes external service information corresponding to the operator identifier.

[0054] In an embodiment of the present invention, operator indicator data may be obtained according to a defined operator indicator template, the operator indicator template includes various fields assembled according to a preset format, each field includes a field indicating an operator identifier, a field indicating external service information, and a field indicating indicator data, and the operator indicator template may support obtaining metrics of calculation results by methods other than Flink, such as relying on external services. Among them, the field indicating the operator identifier may include a field indicating the operator name and a field indicating the operator code, the field indicating the external service information includes a field indicating the external service type, and may also include a field indicating the detailed information of the external service type, and the field indicating the indicator data may include one or more of the operator input data volume, the operator output data volume, the cumulative hit data volume, and the cumulative miss data volume. The preset format may include the arrangement order of the various fields and the connection symbols between the various fields, and the arrangement order of the various fields may be a field indicating the operator name, a field indicating the operator code, a field indicating the external service type, a field indicating the detailed information of the external service type, and a field indicating the indicator data, and the various fields may be connected by ".".

[0055] For example, the operator indicator template in the logistics risk control scenario can be defined as: ${name}.${taskId}.${scope}.${detail}.[input|output|hit|miss]Count, where name is the operator name, taskId is the operator code under the operator name, scope is the external service type, detail is the detailed information of the external service type, and [input|output|hit|miss]Count are the operator input data volume (inputCount), operator output data volume (outputCount), cumulative hit data volume (hitCount), and cumulative miss data volume (missCount).

[0056] In the embodiment of the present invention, receiving operator indicator data includes: receiving operator indicator data regularly reported by a reporter. Before receiving the operator indicator data, it also includes: adding a graph database reporting class, in which the graph database reporting class indicates a method for the reporter to report the operator indicator data to the graph database.

[0057] After defining the operator indicator template, that is, the custom metric, the operator indicator data can be obtained through the operator indicator template. For example, Flink's task manager (TaskManager) can update the operator indicator data in real time according to the received risk control clues, and report the updated operator indicator data to the reporter in real time or periodically. The reporter (Trace Reporter) is used to receive different metric data in the task manager, that is, different operator indicator data. The reporter reports the received operator indicator data in real time or periodically, so that the graph database receives the operator indicator data.

[0058] Flink does not have a built-in reporter for reporting indicator data to a graph database, so a reporter that can report to a graph database can be obtained through extension. A graph database is a non-relational database that corresponds to the relationship between vertices and edges. The relationship between data and data is constructed through nodes and edges to form a graph structure. The graph database can be Neo4j or NebulaGraph, for example, NebulaGraph, which is a distributed and scalable graph database that can accommodate graph databases with billions of vertices and trillions of edges, while providing millisecond-level responses.

[0059] Specifically, a new graph database reporting class (such as NebulaReporterFactory class) is added to Flink, which indicates the method of the reporter reporting operator indicator data to the graph database; a custom reporter Reporter is added to Flink-conf.xml, and after successful compilation, it is placed in the Flink / lib directory. The custom reporter will be initialized in each task manager after the Flink task is started, so that the task manager can report the operator indicator data to the custom reporter, and the custom reporter reports the operator indicator data to the graph database.

[0060] After the custom reporter receives the operator indicator data reported by the task manager, it converts the operator indicator data, that is, it parses the first operator identifier and the indicator data corresponding to the first operator identifier from the operator indicator data, and then obtains the converted operator indicator data according to the set format of the reporter. That is, the operator indicator data received by the graph database is obtained after the reporter converts the operator indicator data of the task manager.

[0061] After receiving the operator indicator data, the operator indicator data is written into the graph database to update the graph structure corresponding to the first operator identifier. Specifically, a vertex write statement can be constructed according to the first operator identifier, and an edge data write statement can be constructed according to the first indicator data. The first operator identifier and the first indicator data are written into the graph database, and it is determined whether there is a vertex corresponding to the first operator identifier in the graph database. If so, the corresponding first edge data is updated according to the first indicator data. If not, the corresponding graph structure is constructed according to the first operator identifier and the first indicator data.

[0062] In the embodiment of the present invention, Figure 2 As shown, before updating the first edge data with the first operator identifier as a vertex in the graph database according to the first indicator data corresponding to the first operator identifier, it includes:

[0063] Step S201: receiving third operator indicator data, where the third operator indicator data includes a first operator identifier and third indicator data corresponding to the first operator identifier;

[0064] Step S202: Acquire a data flow relationship corresponding to the first operator identifier;

[0065] Step S203: construct a graph structure corresponding to the first operator identifier according to the first operator identifier, the third indicator data, and the data flow relationship.

[0066] In an embodiment of the present invention, before updating the first edge data with the first operator identifier as a vertex in the graph database, a graph structure corresponding to the first operator identifier is constructed. Specifically, the third operator indicator data is received, the third operator indicator data includes the first operator identifier and the third indicator data of the first operator identifier, and the data flow relationship corresponding to the first operator identifier is obtained from the configured data topology graph such as the Flink topology graph, and the data flow relationship corresponding to the first operator identifier includes the upstream operator identifier and the downstream operator identifier of the first operator identifier. The graph structure corresponding to the first operator identifier is constructed according to the data flow relationship, the first operator identifier and the third indicator data.

[0067] In an embodiment of the present invention, a graph structure corresponding to the first operator identifier is constructed based on the first operator identifier, the third indicator data and the data flow relationship, including: obtaining the third operator identifier downstream of the first operator identifier from the data flow relationship; using the first operator identifier and the third operator identifier as vertices respectively, and the third indicator data as edge data from the first operator identifier to the third operator identifier, to construct a graph structure.

[0068] In an embodiment of the present invention, when constructing a graph structure corresponding to a first operator identifier, a third operator identifier downstream of the first operator identifier is obtained, that is, the first operator identifier data flows to the third operator identifier, and the vertices corresponding to the first operator identifier and the vertices corresponding to the third operator identifier are respectively constructed, and the first operator identifier and the third operator identifier are connected through an edge, the direction of the edge is from the first operator identifier to the third operator identifier, and the third indicator data is used as the edge data, thereby obtaining the corresponding graph structure. That is, the graph structure includes the vertices corresponding to the first operator identifier and the vertices corresponding to the third operator identifier, as well as the edges and edge data from the vertices of the first operator identifier to the vertices corresponding to the third identifier.

[0069] In an embodiment of the present invention, before constructing a graph structure, each vertex (Vertex) and the edge (Edge) between each vertex are pre-defined in the graph database, and the operator identifier can be defined as a vertex. The vertex data can include the operator name and operator type, and the data type of the operator name and operator type can also be set. Define the data flow direction, that is, the indicator data as an edge. The edge data can include the external service type, the indicator reporting time, the operator input data volume, the operator output data volume, the cumulative hit data volume, and the cumulative miss data volume. The corresponding data type can also be set, and the operator output data volume, the cumulative hit data volume, and the cumulative miss data volume can also be set to allow null values ​​according to different operator types.

[0070] In an embodiment of the present invention, updating the first edge data with the first operator identifier as a vertex in the graph database according to the first indicator data corresponding to the first operator identifier includes: obtaining a graph structure including the vertex corresponding to the first operator identifier, obtaining the first edge data corresponding to the first operator identifier from the graph structure, and updating the first edge data according to the first indicator data. That is, first obtaining a graph structure including the vertex corresponding to the first operator identifier, the graph structure including the vertex corresponding to the first operator identifier and the vertex corresponding to the third operator identifier downstream of the first operator identifier and the edge indicating the data flow direction between the two vertices, obtaining the first edge data corresponding to the vertex from the graph structure, and updating the first edge data using the first indicator data to implement the update of the graph structure.

[0071] In an embodiment of the present invention, in the data monitoring method, each operator identifier and the indicator data corresponding to each operator identifier are stored in the graph database in the graph structure of vertices and edges. The user can send a query request for the target operator identifier to the graph database to query the target indicator data corresponding to the target operator identifier and determine whether the preset conditions of the target operator identifier are met to determine whether to generate warning information. The graph database is NebulaGraph, and the query request can be queried using nGQL, which is a declarative graph query language used by NebulaGraph and supports flexible and efficient graph modes.

[0072] In an embodiment of the present invention, it is possible to determine whether there are data quality problems with operator identifiers and indicator data through a scheduled task, wherein the scheduled task can monitor one or more operator identifiers, and the specific monitored operator identifiers can be customized. The time interval for triggering the scheduled task can be customized, and the monitoring granularity can be set every minute, every hour, every day, etc. For example, the scheduled task can be set to be triggered once every minute, thereby achieving real-time monitoring and active monitoring of data quality.

[0073] The triggering of the scheduled task is monitored, and the scheduled task indicates the second operator identifier, wherein the scheduled task can be triggered by sending a query request at a scheduled time, and the query request can be sent in nGQL. The second operator identifier has a corresponding vertex in the graph database, and the second operator identifier can be one or more, and the second operator identifier can be the first operator identifier, or it can not be the first operator identifier; query the corresponding second edge data from the graph database according to the second operator identifier, obtain the preset condition corresponding to the second operator identifier, and the preset condition can indicate the preset threshold or preset ratio range of the edge data, and judge whether the second edge data meets the preset condition. If it does not meet the preset condition, generate the warning information corresponding to the second operator identifier, and if it meets the preset condition, do not process it. For example, the first edge data includes the operator input data volume and the operator output data volume. The preset condition can be that the ratio of the operator input data volume to the output data volume is within the preset ratio range. If the operator input data volume and the output data volume of the first edge data are not within the preset ratio range, it means that there is a data quality problem in the first edge data, and the corresponding warning information is generated. The warning information can be transmitted to the corresponding staff by phone or text message.

[0074] In an embodiment of the present invention, when generating warning information corresponding to the second operator identifier, the corresponding warning information can be generated based on the second operator identifier; the warning information can also be generated based on the second operator identifier, the second edge data and external service information, that is, if it depends on external service information, then the warning information containing the external service information is generated.

[0075] Figure 3The present invention is a flowchart of a data monitoring method according to an embodiment of the present invention. Each Taskmanager of Flink reports the obtained operator indicator data to the trace reporter, and the trace reporter synchronizes the operator indicator data to the NebulaGraph graph database. The operator indicator data includes a first operator identifier and the first indicator data corresponding to the first operator identifier. The vertex corresponding to the first operator identifier and the first edge data corresponding to the vertex are obtained from the graph database, and the first edge data is updated using the first indicator data. The graph database includes a graph structure constructed with each operator identifier as a vertex and the data flow relationship between each operator identifier as an edge. Studio is an open source graph data block visualization tool in NebulaGraph that can be accessed through the web. The vertices and edges of NebulaGraph can be defined through studio, and nGQL query can also be implemented. For example, users can query whether the first edge data of the first operator identifier meets the preset condition corresponding to the first operator identifier through studio, and can also use nGQL query to determine whether the second edge data of the second operator identifier meets the preset condition of the second operator identifier through a scheduled task. If not, an early warning message is generated to notify the staff. The data monitoring method of the embodiment of the present invention can support data quality monitoring of dynamic judgment logic, with a high troubleshooting efficiency of 0.5~1 man-hour, which is significantly higher than the 1~2 man-hour of the prior art. The data monitoring granularity can be configured to minutes, hours, days, etc., and active monitoring can be achieved.

[0076] The data monitoring method provided by the embodiment of the present invention, after receiving the operator indicator data, updates the first edge data with the first operator identifier as the vertex in the graph database according to the first indicator data corresponding to the first operator identifier of the operator indicator data. After monitoring the departure of the scheduled task, the second edge data corresponding to the second operator identifier indicated by the scheduled task is obtained from the graph database, and when the second edge data does not meet the preset conditions corresponding to the second operator identifier, the warning information corresponding to the second operator identifier is generated. The method achieves the effect of early warning for data quality monitoring by writing the operator indicator data into the graph database and making a real-time judgment of the preset conditions. The graph database can quickly locate data quality problems; the method realizes flexible monitoring of data quality problems by combining the stream processing framework with the graph database, and uniformly monitors the data quality of the dynamic judgment logic that depends on external services by customizing the operator indicator template; the method can realize active monitoring of data quality and improve the efficiency of data quality troubleshooting.

[0077] According to another aspect of the embodiment of the present invention, Figure 4 As shown, a data monitoring device 400 is provided, comprising:

[0078] A receiving module 401 receives operator indicator data, where the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier;

[0079] An updating module 402 updates first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier;

[0080] The acquisition module 403 acquires second edge data corresponding to the second operator identifier from the graph database in response to monitoring the triggering of the scheduled task, where the scheduled task indicates the second operator identifier;

[0081] The generating module 404 generates warning information corresponding to the second operator identifier when the second edge data does not satisfy a preset condition corresponding to the second operator identifier.

[0082] In an embodiment of the present invention, the receiving module 401 is also used to: before receiving the operator indicator data, define an operator indicator template, the operator indicator template includes a field indicating the operator identifier, a field indicating the indicator data, and a field indicating external service information, and the operator indicator data also includes the external service information corresponding to the operator identifier.

[0083] In an embodiment of the present invention, the receiving module 401 is further used to: receive operator indicator data reported periodically by a reporter; the receiving module 401 is also used to: before receiving the operator indicator data, add a graph database reporting class, and the graph database reporting class indicates a method for the reporter to report the operator indicator data to the graph database.

[0084] In an embodiment of the present invention, the update module 402 is also used to: before updating the first edge data with the first operator identifier as the vertex in the graph database according to the first indicator data corresponding to the first operator identifier, receive third operator indicator data, the third operator indicator data including the first operator identifier and the third indicator data corresponding to the first operator identifier; obtain the data flow relationship corresponding to the first operator identifier; and construct a graph structure corresponding to the first operator identifier according to the first operator identifier, the third indicator data and the data flow relationship.

[0085] In an embodiment of the present invention, the update module 402 is further used to: obtain a third operator identifier downstream of the first operator identifier from the data flow relationship; use the first operator identifier and the third operator identifier as vertices respectively, and use the third indicator data as edge data from the first operator identifier to the third operator identifier to construct a graph structure.

[0086] In an embodiment of the present invention, the updating module 402 is further used to: obtain a graph structure including vertices corresponding to the first operator identifier, obtain first edge data corresponding to the first operator identifier from the graph structure, and update the first edge data according to the first indicator data.

[0087] In the embodiment of the present invention, the generating module 404 is further configured to generate warning information according to the second operator identifier, the second edge data and the external service information.

[0088] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data monitoring method provided by the present invention.

[0089] According to another aspect of an embodiment of the present invention, a computer readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data monitoring method provided by the present invention is implemented.

[0090] Figure 5 An exemplary system architecture 500 is shown to which the data monitoring method or data monitoring device according to the embodiment of the present invention can be applied.

[0091] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0092] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0093] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0094] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0095] It should be noted that the data monitoring method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data monitoring device is generally disposed in the server 505 .

[0096] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0097] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0098] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0100] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0101] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a receiving module, an updating module, an acquiring module and a generating module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the receiving module may also be described as a "module for receiving operator indicator data".

[0104] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving operator indicator data, the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier; updating the first edge data with the first operator identifier as the vertex in the graph database according to the first indicator data corresponding to the first operator identifier; in response to monitoring the triggering of a scheduled task, the scheduled task indicates a second operator identifier, obtaining the second edge data corresponding to the second operator identifier from the graph database; generating warning information corresponding to the second operator identifier when the second edge data does not meet the preset condition corresponding to the second operator identifier.

[0105] According to the technical solution of the embodiment of the present invention, the data monitoring method provided by the embodiment of the present invention, after receiving the operator indicator data, updates the first edge data with the first operator identifier as the vertex in the graph database according to the first indicator data corresponding to the first operator identifier of the operator indicator data. After monitoring the departure of the scheduled task, the second edge data corresponding to the second operator identifier indicated by the scheduled task is obtained from the graph database, and when the second edge data does not meet the preset conditions corresponding to the second operator identifier, the warning information corresponding to the second operator identifier is generated. The method achieves the effect of early warning for data quality monitoring by writing the operator indicator data into the graph database and making a real-time judgment of the preset conditions. The graph database can quickly locate data quality problems; the method realizes flexible monitoring of data quality problems by combining the stream processing framework with the graph database, and uniformly monitors the data quality of the dynamic judgment logic that depends on external services through custom metrics; the method can realize active monitoring of data quality and improve the efficiency of data quality troubleshooting.

[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data monitoring method, characterized in that: include: Receive operator indicator data, where the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier; Update first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier; In response to monitoring the triggering of a scheduled task, the scheduled task indicates a second operator identifier, acquiring second edge data corresponding to the second operator identifier from the graph database; When the second edge data does not satisfy a preset condition corresponding to the second operator identifier, a warning message corresponding to the second operator identifier is generated.

2. The method according to claim 1, characterized in that Before receiving operator indicator data, including: An operator indicator template is defined, wherein the operator indicator template includes a field indicating an operator identifier, a field indicating indicator data, and a field indicating external service information, wherein the operator indicator data also includes external service information corresponding to the operator identifier.

3. The method according to claim 1, characterized in that: Receiving operator indicator data, including: receiving the operator indicator data regularly reported by a reporter; Before receiving operator indicator data, it also includes: A new graph database reporting class is added, in which the graph database reporting class indicates the method by which the reporter reports the operator indicator data to the graph database.

4. The method according to claim 1, characterized in that: Before updating first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier, the method includes: receiving third operator indicator data, wherein the third operator indicator data includes a first operator identifier and third indicator data corresponding to the first operator identifier; Acquire a data flow relationship corresponding to the first operator identifier; A graph structure corresponding to the first operator identifier is constructed according to the first operator identifier, the third indicator data, and the data flow relationship.

5. The method according to claim 4, characterized in that Constructing a graph structure corresponding to the first operator identifier according to the first operator identifier, the third indicator data, and the data flow relationship, including: Acquire a third operator identifier downstream of the first operator identifier from the data flow relationship; The graph structure is constructed by taking the first operator identifier and the third operator identifier as vertices respectively and taking the third indicator data as edge data pointing from the first operator identifier to the third operator identifier.

6. The method according to claim 4, characterized in that Updating first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier includes: A graph structure including vertices corresponding to the first operator identifier is obtained, first edge data corresponding to the first operator identifier is obtained from the graph structure, and the first edge data is updated according to the first indicator data.

7. The method according to claim 2, characterized in that: Generating warning information corresponding to the second operator identifier includes: The warning information is generated according to the second operator identifier, the second edge data, and the external service information.

8. A data monitoring device, characterized in that: include: A receiving module receives operator indicator data, where the operator indicator data includes a first operator identifier and first indicator data corresponding to the first operator identifier; An updating module, which updates first edge data with the first operator identifier as a vertex in a graph database according to first indicator data corresponding to the first operator identifier; an acquisition module, in response to monitoring the triggering of a scheduled task, wherein the scheduled task indicates a second operator identifier, acquiring second edge data corresponding to the second operator identifier from the graph database; A generating module is configured to generate warning information corresponding to the second operator identifier when the second edge data does not satisfy a preset condition corresponding to the second operator identifier.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images