Data monitoring method, device and electronic equipment

Abstract

The present application provides a data monitoring method, device and electronic device, which relate to the field of computer technology. The data monitoring method is applied to the monitoring equipment of the data processing system, and the data processing system also includes a business service device. The method includes obtaining the operation data of the target business data, the target business data is the business data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system, and the operation data includes at least: the service identifier of the target service, the consumption time of the target business data and the event time of the target business data; calculate the difference between the consumption time and the event time to obtain the consumption delay of the target business data; when the consumption delay is greater than the delay threshold corresponding to the target service, generate an alarm information of the target business data. The present application improves the accuracy of delay monitoring and improves the delay monitoring effect.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data monitoring method, device and electronic equipment. Background Art

[0002] Apache Flink is an open source computing framework for distributed data stream processing and batch data processing, which is widely used in the field of data processing. Apache Flink is referred to as Flink, and its core is a distributed stream data computing engine. Kafka is a distributed message queue based on the publish / subscribe mode, which is mainly used in the field of real-time data processing. In the data processing system that combines Flink and Kafka, Flink can act as a consumer of the Kafka message queue, read messages from the Kafka message queue, and further process the read messages. However, there may be a delay between the generation and consumption of messages in the Kafka message queue, which affects the processing effect of the message. Therefore, a data monitoring solution is urgently needed to monitor whether the Flink system has a consumption delay on the messages in the Kafka message queue.

[0003] In the current data monitoring solution, metric data is obtained through the metric data interface of the Flink system. The metric data includes the consumption offset of the Kafka message queue and the total offset of the Kafka message queue. Among them, the consumption offset is used to reflect the number of messages consumed by the consumer in the Kafka message queue. The total offset is used to reflect the total number of messages that the consumer can consume in the Kafka message queue. By determining whether the consumption offset is consistent with the total offset, when the consumption offset is inconsistent with the total offset, it is determined that there are unconsumed messages in the Kafka message queue, and then the consumption delay is determined.

[0004] However, the current data monitoring solution can only monitor whether consumption delay occurs for messages in the Kafka message queue, but cannot locate the specific data where consumption delay occurs, and the delay monitoring effect is poor. Summary of the invention

[0005] In view of this, the present application provides a data monitoring method, device, electronic device and storage medium, which improve the accuracy of delay monitoring to a certain extent and improve the delay monitoring effect.

[0006] According to a first aspect of the present application, a data monitoring method is provided, which is applied to a monitoring device of a data processing system, wherein the data processing system further includes a business service device, and the business service device runs a Flink system, wherein the method includes:

[0007] Acquire operation data of target business data, where the target business data is business data of the target service in the first Kafka message queue consumed when the business service device executes the target service through the Flink system, and the operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data;

[0008] Calculate the difference between the consumption time and the event time to obtain the consumption delay of the target business data;

[0009] When the consumption delay is greater than the delay threshold corresponding to the target service, alarm information of the target service data is generated.

[0010] According to a second aspect of the present application, a data monitoring method is provided, which is applied to a business service device of a data processing system, wherein the business service device runs a Flink system, and the data processing system further includes a monitoring device, and the method includes:

[0011] Receive execution instructions from the target service;

[0012] Read the target business data of the target service in the first kafka message queue through the Flink system;

[0013] Generate operation data of the target business data, the operation data at least including: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data;

[0014] The operation data is sent to the monitoring device so that the monitoring device calculates the difference between the consumed time and the event time after acquiring the operation data, and obtains the consumption delay of the target business data. When the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated.

[0015] According to a third aspect of the present application, a data monitoring device is provided, which is applied to a monitoring device of a data processing system, wherein the data processing system further includes a business service device, and the business service device runs a Flink system, wherein the device includes:

[0016] An acquisition module is used to acquire operation data of target business data, where the target business data is data of the target service in the first Kafka message queue consumed when the business service device executes the target service through the Flink system, and the operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data;

[0017] A calculation module, used to calculate the difference between the consumption time and the event time to obtain the consumption delay of the target business data;

[0018] A generation module is used to generate an alarm message when the consumption delay is greater than the delay threshold corresponding to the target service.

[0019] According to a fourth aspect of the present application, a data monitoring device is provided, which is applied to a business service device of a data processing system, wherein the business service device runs a Flink system, and the data processing system further includes a monitoring device, wherein the device includes:

[0020] A receiving module, used for receiving an execution instruction of a target service;

[0021] A reading module, used to read the target business data of the target service in the first Kafka message queue through the Flink system;

[0022] A generating module, configured to generate operation data of the target service data, wherein the operation data at least includes: a service identifier of the target service, a consumption time of the target service data, and an event time of the target service data;

[0023] A sending module is used to send the operation data to the monitoring device, so that the monitoring device calculates the difference between the consumed time and the event time after acquiring the operation data, obtains the consumption delay of the target business data, and generates an alarm message when the consumption delay is greater than the delay threshold corresponding to the target service.

[0024] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data monitoring method as described in either the first aspect or the second aspect are implemented.

[0025] In a sixth aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data monitoring method as described in either the first aspect or the second aspect are implemented.

[0026] In a seventh aspect, an embodiment of the present application provides a data processing system, the data processing system comprising: a monitoring device and a business service device. The monitoring device is used to implement the steps of the data monitoring method as described in any one of the first aspects. The business service device is used to implement the steps of the data monitoring method as described in any one of the second aspects.

[0027] This application has the following advantages:

[0028] The data monitoring method, device and electronic device provided in the embodiment of the present application can obtain the operation data of the target business data of the target service, and the target business data is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data and the event time of the target business data. The consumption delay of the target business data is obtained by calculating the difference between the consumption time and the event time of the target business data. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0030] Figure 2 is a structural diagram of another data processing system provided in an embodiment of the present application;

[0031] Figure 3 is a flow chart of a data monitoring method provided by an embodiment of the present application;

[0032] Figure 4 is a flow chart of a data monitoring method provided by an embodiment of the present application;

[0033] Figure 5 is a flow chart of a data monitoring method provided by an embodiment of the present application;

[0034] Figure 6 is a schematic diagram of a related data query interface provided in an embodiment of the present application;

[0035] Figure 7 is a flow chart of a data monitoring method provided by an embodiment of the present application;

[0036] Figure 8 is a block diagram of a data monitoring device provided in an embodiment of the present application;

[0037] Fig. 9 is a block diagram of a data monitoring device provided in an embodiment of the present application;

[0038] Fig.10 It is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Please refer to Figure 1 , which shows a schematic diagram of the structure of a data processing system provided by an embodiment of the present application. Figure 1 As shown, the data processing system 100 includes: a monitoring device 101 and a business service device 102. The business service device 102 runs a Flink system.

[0041] Among them, the business service device running the Flink system can communicate with the first Kafka message queue to read the business data in the first Kafka message queue as a consumer of the first Kafka message queue. The first Kafka message queue stores the business data of multiple business services provided by the business service device. The business data of the business service can be provided by the data production end. Optionally, the first Kafka message queue can run on the business service device. Alternatively, the first Kafka message queue can run on a third-party device.

[0042] In an optional implementation, the monitoring device and the business service device can be connected via a wired network or a wireless network. The business service device can send data directly to the monitoring device through the network between it and the monitoring device. Among them, the wired network is such as a Universal Serial Bus (USB), and the wireless network includes: a cellular network, a wireless fidelity (WIREless FIdelity, WIFI) network, a Bluetooth network, an infrared network or a Zigbee network. For example, the cellular network can be a Long Term Evolution (LTE) network or a 5G network.

[0043] In another optional implementation, the monitoring device and the business service device can both communicate with the second Kafka message queue. The business service device can write the data to be sent to the monitoring device into the second Kafka message queue as a producer. The monitoring device can read the data transmitted by the business service device from the second Kafka message queue as a consumer. That is, the monitoring device and the business service device can communicate through the second Kafka message queue.

[0044] In some embodiments of the present application, the business service device and the monitoring device may be a computer, a mobile phone, a wearable device, one or more servers, or a cloud service cluster, etc. Figure 1 For example, the business service device is a computer and the monitoring device is a computer.

[0045] Alternatively, if Figure 2 As shown, in Figure 1 Based on the data transmission system 100 shown, the data transmission system 100 may further include: a monitoring personnel terminal 103. The monitoring personnel terminal may be connected to the monitoring device via a wired network or a wireless network. The monitoring personnel terminal may receive an alarm message sent by the monitoring device, and the alarm message is used to remind the monitoring personnel that the consumption delay of the business data of the business service has timed out. For example, the monitoring personnel terminal may be a computer, a mobile phone, a tablet computer, or a wearable device. Figure 2 For example, the business service device is a computer, the monitoring device is a computer, and the monitoring personnel terminal is a mobile phone.

[0046] Please refer to Figure 3 , which shows a flow chart of a data monitoring method provided by an embodiment of the present application. The data monitoring method can be applied to Figure 1 or Figure 2 The data processing system shown in FIG. Figure 3 As shown, the data monitoring method includes:

[0047] Step 301: The business service device receives an execution instruction of a target service.

[0048] In an embodiment of the present application, the business service device can provide a variety of business services. The target service can be one of the multiple business services. Optionally, a business user of the business service device can input an execution operation of the target service to the business service device, so that after receiving the execution operation of the target service, the business service device generates an execution instruction of the target service in response to the execution operation. Alternatively, the business service device generates an execution instruction of the target service after the execution trigger condition of the target service is met. Among them, the execution instruction of the target service may include a service identifier of the target service. The service identifier is used to uniquely identify the business service.

[0049] Step 302: The business service device reads the target business data of the target service in the first Kafka message queue through the Flink system.

[0050] In the embodiment of the present application, the Flink system running on the business service device can act as a consumer to read the target business data of the target service in the first Kafka message queue. That is, the Flink system running on the business service device consumes the target business data of the target service in the first Kafka message queue.

[0051] For example, the business service device can use the kafka-source-connector component through the Flink system to read the target business data of the target service in the first kafka message queue.

[0052] Step 303: The business service device generates operation data of the target business data. The operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data.

[0053] In the embodiment of the present application, the business data of the business service stored in the first Kafka message queue includes the event time of the business service. Event time refers to the time when the business data of the business service occurs at its generating end, that is, the time when the business data of the business service is generated at its generating end. Consumption time refers to the time when the business data of the business service is read by the business service device through the Flink system.

[0054] The business service device may record the consumption time of the target business data after reading the target business data of the target service in the first Kafka message queue through the Flink system.

[0055] In an optional implementation, the business service device may parse the target business data of the target service and extract the event time of the target business data from the target business data. Then, the business service device generates the operation data of the target business data according to the recorded consumption time of the target business data, the event time of the extracted target business data, and the service identifier of the target service.

[0056] In another optional implementation, since the target business data of the target service includes the event time of the target business data. Therefore, the operation data of the target business data may include: the target business data, the service identifier of the target service, and the consumption time of the target business data. Among them, the target business data includes the event time of the target business data. In this way, in the process of generating the operation data of the target business data, there is no need to further analyze the target business data to extract the event time of the target business data, which simplifies the generation steps of the operation data and improves the generation efficiency.

[0057] In an embodiment of the present application, the operation data of the target business data may further include: a data identifier of the target business data. The data identifier is used to uniquely identify the business data. In this way, the subsequent monitoring device can identify the business data to which the operation data belongs after acquiring the operation data. Optionally, the data identifier of the target business data may be the target business data (i.e., the original data of the target business data). Alternatively, the data identifier of the target business data may be the offset of the target business data for the first Kafka message queue.

[0058] For example, the business service device can generate the operation data of the target business data after reading the target business data of the target service in the first kafka message queue using the kafka-source-connector component through the Flink system. The kafka-source-connector component can be used to read the business data from the first kafka message queue and generate the operation data of the read business data. The implementation process of generating the operation data of the target business data can be one of the two aforementioned implementation methods.

[0059] Step 304: The business service device sends operation data to the monitoring device.

[0060] As mentioned above, there are many situations in which the business service device and the monitoring device communicate with each other. The embodiment of the present application further explains the process of the business service device sending operation data to the monitoring device in the above-mentioned various situations.

[0061] In the first optional case, the business service device and the monitoring device are connected via a network. The process of the business service device sending operation data to the monitoring device includes: the business service device can directly send the operation data of the target business data of the target service to the monitoring device via the network connected between the business service device and the monitoring device.

[0062] In the second optional case, the business service device and the monitoring device communicate through the second Kafka message queue.

[0063] In an optional implementation, the process of the business service device sending operation data to the monitoring device may include: the business service device sends the operation data of the target business data to the second Kafka message queue, so that the monitoring device obtains the operation data of the target business data from the second Kafka message queue.

[0064] Among them, the business service device can send the operation data of the target business data to the second Kafka message queue as a producer, so that the monitoring device obtains the operation data of the target business data from the second Kafka message queue as a consumer.

[0065] For example, the business service device sends the operation data of the target business data to the second Kafka message queue through the Kafka-Connect component. The Kafka-Connect component is a tool for streaming data between the Kafka message queue and other systems. The Kafka-Connect component can move a large amount of data into or out of the Kafka message queue.

[0066] In another optional implementation, the business service device stores a consumption data file. The consumption data file is used to store the operation data of the business data of the business service in the first kafka message queue consumed by the business service device when executing the business service through the Flink system. The business service device can use the consumption data file to collect the operation data of the business data of each business service, which facilitates the batch management of the operation data of each business data. For example, the consumption data file can be a log file. For example, the consumption data file is a file.log file. Or, the consumption data file can be a DataConsumption.log file.

[0067] Before step 304, the method further includes the following steps 001 and 002.

[0068] In step 001, the business service device stores operation data of target business data of a target service in a consumption data file.

[0069] Optionally, the business service device can use a log output component to collect operation data and store the operation data in a consumption data file. The log output tool is used to collect data, generate and manage logs. For example, the log output component can be logback or log4j2.

[0070] In an exemplary implementation, the log output component is a subcomponent of the kafka-source-connector component. The business service device can use the log output component in the kafka-source-connector component to collect the operation data of the target business data and store the operation data in the consumption data file.

[0071] In step 002, the business service device monitors the consumption data file.

[0072] In the embodiment of the present application, the business service device can monitor the consumption data file in real time, and when it monitors that the consumption data file stores new operation data, it sends the new operation data to the monitoring device. Then the monitoring device determines the consumption delay of the new operation data. The implementation method of the monitoring device determining the consumption delay of the operation data refers to the subsequent steps 305 to 307, and the embodiment of the present application is not described in detail here.

[0073] Optionally, the business service device may monitor consumption data files stored in a specified directory. For example, the business service device may monitor consumption data files using a file collection tool. The file collection tool is used to monitor a specified directory or a specified file, track the data update of the read file, and forward the file data. For example, the file collection tool may be filebeta.

[0074] Based on this, the process of the business service device sending operation data to the monitoring device may include: when it is monitored that the consumption data file stores the operation data of the target business data, sending the operation data of the target business data to the second Kafka message queue, so that the monitoring device obtains the operation data of the target business data from the second Kafka message queue.

[0075] For example, when the business service device uses the file collection tool to monitor that the consumption data file stores the operation data of the target business data, the business service device sends the operation data of the target business data to the second Kafka message queue.

[0076] Step 305: The monitoring device obtains operation data of the target business data.

[0077] In the embodiment of the present application, the communication between the aforementioned business service device and the monitoring device is in the first case, that is, the business service device and the monitoring device are connected via a network. The process of the monitoring device acquiring the operation data of the target business data includes: the monitoring device can receive the operation data of the target business data sent by the business service device to acquire the operation data of the target business data. In some optional embodiments, the monitoring device can also read the operation data of the target business data from the business service device to acquire the operation data of the target business data.

[0078] The second case of communication between the aforementioned business service device and the monitoring device is adopted, that is, the business service device and the monitoring device communicate through the second Kafka message queue. The process of the monitoring device obtaining the operation data of the target business data includes: the monitoring device reads the operation data of the target business data from the second Kafka message queue to obtain the operation data of the target business data.

[0079] Step 306: The monitoring device calculates the difference between the consumption time and the event time to obtain the consumption delay of the target business data.

[0080] In the embodiment of the present application, the monitoring device calculates the difference between the consumption time and the event time of the target business data to obtain the consumption delay of the target business data.

[0081] Step 307: When the consumption delay is greater than the delay threshold corresponding to the target service, the monitoring device generates alarm information for the target service data.

[0082] In the embodiment of the present application, after obtaining the consumption delay of the target service data, the monitoring device can compare the consumption delay with the delay threshold corresponding to the target service. When the consumption delay of the target service data is greater than the delay threshold corresponding to the target service, it indicates that the consumption delay of the target service data is high, and an alarm message is generated. The alarm message is used to prompt that the consumption delay of the target service data of the target service is high.

[0083] Among them, the delay threshold corresponding to the target service can be understood as the maximum consumption delay of the target business data that can be tolerated when the business service device executes the target service. The delay thresholds corresponding to multiple business services provided by the business service device can be the same or different. And the value of the delay threshold corresponding to the business service can be determined according to the service requirements of the business service in actual circumstances. Optionally, the delay threshold corresponding to the target service can be set and input by the user. Alternatively, the delay threshold corresponding to the target service can be calculated and determined by the monitoring device based on the service performance of the target service.

[0084] Optionally, the monitoring device may store a service delay profile. The service delay profile records delay thresholds corresponding to service identifiers of multiple business services. Before step 307, the method further includes: searching the service delay profile for a delay threshold corresponding to the service identifier of the target service. For example, the service delay profile stores a correspondence between the service identifier and the delay threshold. The monitoring device may determine the delay threshold corresponding to the service identifier of the target service from the correspondence.

[0085] In some embodiments of the present application, the alarm information generated by the monitoring device when the consumption delay is greater than the delay threshold corresponding to the target service can be displayed on the monitoring device. Alternatively, the alarm information of the monitoring device can be sent to the monitoring personnel terminal. The method also includes: the monitoring device sends the alarm information of the target business data to the monitoring personnel terminal. The alarm information includes the data identifier of the target business data, the service identifier of the target service, and the consumption delay of the target business data. Optionally, the monitoring device sends an instant message to the monitoring personnel terminal, and the instant message includes the alarm information.

[0086] In summary, the data monitoring method provided in the embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, and the target business data is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. By calculating the difference between the consumption time and the event time of the target business data, the consumption delay of the target business data is obtained. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data where consumption delay occurs, and alarm the business data where consumption delay occurs, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0087] Please refer to Figure 4 , which shows a flow chart of another data monitoring method provided by an embodiment of the present application. The data monitoring method can be applied to Figure 1 or Figure 2 The data processing system shown in FIG. Figure 4 As shown, the data monitoring method includes:

[0088] Step 401: The business service device receives an execution instruction of a target service.

[0089] Step 402: The business service device reads the target business data of the target service in the first Kafka message queue through the Flink system.

[0090] Step 403: The business service device generates operation data of the target business data. The operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data.

[0091] Step 404: The business service device stores the operation data of the target business data of the target service in the consumption data file.

[0092] Step 405: The business service device monitors the consumption data file.

[0093] Step 406: When the business service device monitors that the consumption data file stores the operation data of the target business data, the business service device sends the operation data of the target business data to the second Kafka message queue.

[0094] Step 407: The monitoring device reads the operation data of the target business data from the second Kafka message queue.

[0095] Step 408: The monitoring device calculates the difference between the consumption time and the event time to obtain the consumption delay of the target business data.

[0096] Step 409: When the consumption delay is greater than the delay threshold corresponding to the target service, the monitoring device generates alarm information for the target service data.

[0097] Step 410: The monitoring device sends an alarm message to the monitoring personnel terminal, wherein the alarm message includes the data identifier of the target service data, the service identifier of the target service, and the consumption delay of the target service data.

[0098] In the embodiment of the present application, the monitoring device sends the alarm information to the monitoring personnel terminal through the connection network between the monitoring device and the monitoring personnel terminal. Optionally, the monitoring device sends an instant message to the monitoring personnel terminal, and the instant message includes the alarm information.

[0099] The explanation and implementation of steps 401 to 403 and steps 407 to 409 in this embodiment can refer to the explanation and implementation of steps 301 to 303 and steps 305 to 307 in the above embodiment. The explanation and implementation of steps 404 to 406 can refer to the relevant explanation and implementation in the above step 304, which will not be described in detail in this embodiment of the application.

[0100] In an embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, which is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. By calculating the difference between the consumption time and the event time of the target business data, the consumption delay of the target business data is obtained. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0101] For ease of understanding, the data monitoring method provided in the embodiment of the present application is further described by taking the following example as an example. Figure 5 As shown, it shows a flow chart of another data monitoring method provided by an embodiment of the present application. The data monitoring method can be applied to Figure 1 or Figure 2 The data processing system shown.

[0102] When the business service device executes the user job (i.e., the target service), it consumes data from the first kafka message queue using the kafka-source-connector component through the Flink system, and the consumed data is the target business data of the user job. The business service device uses the logback component to output the log file, and the output log file is the consumption data file. The content of the output log file is the operation data of the target business data of the user job. The operation data includes the service identifier of the user job, the target business data of the user job, and the consumption time of the business data. The business service device uses the filebeta component (file collection tool) to monitor the log file. When it is detected that the log file stores new operation data, the new data in the log file is collected, and the collected new data is transmitted to the second kafka message queue. The collected new data is the operation data of the target business data of the user job. When the monitoring device executes the monitoring job, it consumes data from the second kafka message queue to obtain the operation data of the target business data of the user job. The operation data is parsed to obtain the data identifier of the target business data, the service identifier of the user job, the event time of the target business data, and the consumption time, and the consumption delay of the target business data is calculated. The monitoring device reads the threshold configuration data (i.e., the service threshold configuration file) to obtain the delay threshold corresponding to the user job. It determines whether the calculated consumption delay of the target business data is greater than the delay threshold corresponding to the user job. When the consumption delay of the target business data is greater than the delay threshold corresponding to the user job, an alarm message of the target business data is sent to the monitoring personnel terminal. The alarm message may include: the service identifier of the user job and the data identifier of the target business data. Figure 5 The first Kafka is used to identify the first Kafka message queue, and the second Kafka is used to identify the second Kafka message queue.

[0103] In an embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, which is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. By calculating the difference between the consumption time and the event time of the target business data, the consumption delay of the target business data is obtained. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0104] Please refer to Figure 6 , which shows a flow chart of a data monitoring method provided by an embodiment of the present application. The data monitoring method can be applied to Figure 1 or Figure 2 The data processing system shown is executed by the monitoring device. Figure 6 As shown, the data monitoring method includes:

[0105] Step 601: Obtain the operation data of the target business data. The target business data is the business data of the target service in the first Kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data at least includes: the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data.

[0106] Step 602: Calculate the difference between the consumption time and the event time to obtain the consumption delay of the target business data.

[0107] Step 603: When the consumption delay is greater than the delay threshold corresponding to the target service, generate alarm information for the target service data.

[0108] Optionally, obtaining operation data of the target business data includes: reading operation data from a second Kafka message queue, where the operation data in the second Kafka message queue is generated by the business service device after consuming the target business data in the first Kafka message queue and sent to the second Kafka message queue.

[0109] Optionally, when the consumption delay is greater than the delay threshold corresponding to the target service, before generating alarm information for the target business data, the method also includes: searching for the delay threshold corresponding to the service identifier of the target service from a service delay configuration file, the service delay configuration file recording the delay thresholds corresponding to the service identifiers of multiple business services, respectively, and the multiple business services include the target service.

[0110] Optionally, the data processing system also includes a monitoring personnel terminal; the method also includes: sending alarm information to the monitoring personnel terminal, the alarm information including: the data identifier of the target business data, the service identifier of the target service and the consumption delay of the target business data.

[0111] In an embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, which is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. By calculating the difference between the consumption time and the event time of the target business data, the consumption delay of the target business data is obtained. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0112] Please refer to Figure 7 , which shows a flow chart of a data monitoring method provided by an embodiment of the present application. The data monitoring method can be applied to Figure 1 or Figure 2 The data processing system shown is executed by the business service device. Figure 7 As shown, the data monitoring method includes:

[0113] Step 701: Receive an execution instruction of a target service.

[0114] Step 702: Read the target business data of the target service in the first Kafka message queue through the Flink system.

[0115] Step 703: Generate operation data of the target business data, where the operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data.

[0116] Step 704: Send the operation data to the monitoring device so that the monitoring device calculates the difference between the consumption time and the event time after obtaining the operation data, and obtains the consumption delay of the target business data. When the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated.

[0117] Optionally, sending the operation data to the monitoring device includes: sending the operation data to a second Kafka message queue, so that the monitoring device reads the operation data from the second Kafka message queue.

[0118] Optionally, the method further comprises:

[0119] Store the operation data in the consumption data file, which is used to store the operation data of the business data of the business service in the first Kafka message queue consumed by the business service device when executing the business service through the Flink system;

[0120] Monitor consumption data files;

[0121] Sending the operation data to the second Kafka message queue includes: when monitoring that the consumption data file stores the operation data, sending the operation data to the second Kafka message queue.

[0122] In an embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, which is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. By calculating the difference between the consumption time and the event time of the target business data, the consumption delay of the target business data is obtained. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0123] Please refer to Figure 8, which shows a block diagram of a data monitoring device provided in an embodiment of the present application. The data monitoring device is applied to a monitoring device of a data processing system, and the data processing system also includes a business service device, and the business service device runs a Flink system. Figure 8 As shown, the data monitoring device 800 includes: an acquisition module 801 , a calculation module 802 and a generation module 803 .

[0124] The acquisition module 801 is used to acquire the operation data of the target business data. The target business data is the business data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data at least includes: the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data.

[0125] The calculation module 802 is used to calculate the difference between the consumption time and the event time to obtain the consumption delay of the target business data.

[0126] The generating module 803 is used to generate alarm information of the target service data when the consumption delay is greater than the delay threshold corresponding to the target service.

[0127] Optionally, the acquisition module 801 is also used to read operation data from the second Kafka message queue, where the operation data in the second Kafka message queue is generated by the business service device after consuming the target business data in the first Kafka message queue and sent to the second Kafka message queue.

[0128] Optionally, the data monitoring device 800 further includes:

[0129] The search module is used to search for a delay threshold corresponding to a service identifier of a target service from a service delay configuration file, wherein the service delay configuration file records delay thresholds corresponding to service identifiers of multiple business services, including the target service.

[0130] Optionally, the data processing system also includes a monitoring personnel terminal; the data monitoring device 800 also includes: a sending module for sending alarm information to the monitoring personnel terminal, the alarm information including: the data identifier of the target business data, the service identifier of the target service and the consumption delay of the target business data.

[0131] In an embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, which is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. By calculating the difference between the consumption time and the event time of the target business data, the consumption delay of the target business data is obtained. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0132] Please refer to Fig. 9 , which shows a block diagram of another data monitoring device provided in an embodiment of the present application. The data monitoring device is applied to a business service device of a data processing system, the business service device runs a Flink system, and the data processing system also includes a monitoring device. Fig. 9 As shown, the data monitoring device 900 includes: a receiving module 901 , a reading module 902 , a generating module 903 and a sending module 904 .

[0133] The receiving module 901 is used to receive an execution instruction of a target service.

[0134] The reading module 902 is used to read the target business data of the target service in the first Kafka message queue through the Flink system.

[0135] The generating module 903 is used to generate the operation data of the target business data, and the operation data at least includes: the service identifier of the target service, the consumption time of the target business data and the event time of the target business data.

[0136] The sending module 904 is used to send the operation data to the monitoring device, so that the monitoring device calculates the difference between the consumption time and the event time after obtaining the operation data, and obtains the consumption delay of the target business data. When the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated.

[0137] Optionally, the sending module 904 is further used to send operation data to the second Kafka message queue, so that the monitoring device reads the operation data from the second Kafka message queue.

[0138] Optionally, the sending module 904 is further used to store operation data in a consumption data file, where the consumption data file is used to store operation data of business data of the business service in the first kafka message queue consumed by the business service device when executing the business service through the Flink system;

[0139] The data monitoring device 900 further includes: a monitoring module for monitoring consumption data files;

[0140] The sending module 904 is further used to send the operation data to the second Kafka message queue when it is monitored that the consumption data file stores the operation data.

[0141] In an embodiment of the present application, the monitoring device can obtain the operation data of the target business data of the target service, and the target business data is the data of the target service in the first kafka message queue consumed by the business service device when executing the target service through the Flink system. The operation data of the target business data includes the service identifier of the target service, the consumption time of the target business data, and the event time of the target business data. The consumption delay of the target business data is obtained by calculating the difference between the consumption time and the event time of the target business data. Then, when the consumption delay is greater than the delay threshold corresponding to the target service, an alarm information of the target business data is generated. In this technical solution, since the monitoring device can obtain the operation data of each business data consumed by the business service device in the first kafka message queue. Therefore, the monitoring device can obtain the consumption delay of each business data according to the operation data of each business data, and then determine whether it is necessary to send an alarm information for each business data. Therefore, the data monitoring method provided in the embodiment of the present application can specifically locate the business data with consumption delay, and alarm the business data with consumption delay, improve the accuracy of delay monitoring, and improve the delay monitoring effect.

[0142] The data monitoring device provided in the embodiment of the present application has a functional module corresponding to the execution of the data monitoring method, and can execute the data monitoring method provided in any embodiment of the present application and achieve the same beneficial effects.

[0143] In one embodiment provided in the present application, a data monitoring system is also provided. The data monitoring system can be Figure 1 or Figure 2The data monitoring system shown. The data monitoring system includes a monitoring device and a business service device. The monitoring device can execute the steps executed by the monitoring device in the data monitoring method provided in any embodiment of the present application. The business service device can execute the steps executed by the business service device in the data monitoring method provided in any embodiment of the present application.

[0144] In another embodiment provided in the present application, an electronic device is also provided. Fig.10 As shown, the electronic device 1000 may include: a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the program, each process of the steps performed by the monitoring device in the data monitoring method provided in any embodiment of the present application is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Alternatively, when the processor 1001 executes the program, each process of the steps performed by the business service device in the data monitoring method provided in any embodiment of the present application is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0145] In another embodiment provided by the present application, a computer-readable storage medium is provided, in which instructions are stored, and when the instructions are executed on a computer, the computer executes the steps executed by the monitoring device in any of the data monitoring methods described in the above embodiments. Alternatively, the computer executes the steps executed by the business service device in any of the data monitoring methods described in the above embodiments.

[0146] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps executed by the monitoring device in any of the data monitoring methods described in the above embodiments. Alternatively, enables the computer to execute the steps executed by the business service device in any of the data monitoring methods described in the above embodiments.

[0147] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0148] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0150] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0151] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0157] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data monitoring method, characterized in that: A monitoring device applied to a data processing system, wherein the data processing system further includes a business service device, wherein the business service device runs a Flink system, and the method includes: Acquire operation data of target business data, wherein the target business data is the business data of the target service in the first Kafka message queue consumed when the business service device executes the target service through the Flink system, and the operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data; the target service is one of multiple business services, and the target business data is one of multiple business data in the first Kafka message queue consumed by the business service device; the consumption time is the moment when the business data of the business service is read by the business service device through the Flink system; Calculate the difference between the consumption time and the event time to obtain the consumption delay of the target business data, so as to obtain the consumption delay of each business data consumed by the business service device, and determine whether it is necessary to send an alarm message for each business data; When the consumption delay is greater than the delay threshold corresponding to the target service, alarm information of the target service data is generated to locate the service data where consumption delay occurs.

2. The method according to claim 1, characterized in that: The operation data of obtaining target business data includes: The operation data is read from the second Kafka message queue, where the operation data in the second Kafka message queue is generated by the business service device after consuming the target business data in the first Kafka message queue and sent to the second Kafka message queue.

3. The method according to claim 1, characterized in that When the consumption delay is greater than the delay threshold corresponding to the target service, before generating the alarm information of the target service data, the method further includes: A delay threshold corresponding to the service identifier of the target service is searched from a service delay configuration file, wherein the service delay configuration file records delay thresholds respectively corresponding to the service identifiers of multiple business services, and the multiple business services include the target service.

4. The method according to any one of claims 1 to 3, characterized in that: The data processing system also includes a monitoring personnel terminal; the method also includes: The alarm information is sent to the monitoring personnel terminal, where the alarm information includes: the data identifier of the target service data, the service identifier of the target service, and the consumption delay of the target service data.

5. A data monitoring method, characterized in that: A business service device applied to a data processing system, wherein the business service device runs a Flink system, and the data processing system further includes a monitoring device. The method includes: Receiving an execution instruction of a target service; the target service is one of multiple business services; Reading target business data of the target service in the first Kafka message queue through the Flink system; the target business data is one of the multiple business data in the first Kafka message queue consumed by the business service device; Generate operation data of the target business data, the operation data at least including: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data; the consumption time is the moment when the business data of the business service is read by the business service device through the Flink system; The operation data is sent to the monitoring device, so that the monitoring device calculates the difference between the consumed time and the event time after acquiring the operation data, and obtains the consumption delay of the target business data, so as to obtain the consumption delay of each business data consumed by the business service device, and determines whether it is necessary to send an alarm message for each business data. When the consumption delay is greater than the delay threshold corresponding to the target service, an alarm message for the target business data is generated to locate the business data where consumption delay occurs.

6. The method according to claim 5, characterized in that The sending the operation data to the monitoring device comprises: The operation data is sent to a second Kafka message queue so that the monitoring device reads the operation data from the second Kafka message queue.

7. The method according to claim 6, characterized in that The method further comprises: Storing the operation data in a consumption data file, where the consumption data file is used to store the operation data of the business data of the business service in the first kafka message queue consumed by the business service device when executing the business service through the Flink system; Monitoring the consumption data file; The sending the operation data to the second Kafka message queue includes: when it is monitored that the consumption data file stores the operation data, sending the operation data to the second Kafka message queue.

8. A data monitoring device, characterized in that: A monitoring device applied to a data processing system, wherein the data processing system further includes a business service device, wherein the business service device runs a Flink system, and the device includes: An acquisition module is used to acquire operation data of target business data, wherein the target business data is the business data of the target service in the first Kafka message queue consumed when the business service device executes the target service through the Flink system, and the operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data; the target service is one of multiple business services, and the target business data is one of multiple business data in the first Kafka message queue consumed by the business service device; the consumption time is the moment when the business data of the business service is read by the business service device through the Flink system; A calculation module, used to calculate the difference between the consumption time and the event time, obtain the consumption delay of the target business data, so as to obtain the consumption delay of each business data consumed by the business service device, and determine whether it is necessary to send an alarm message for each business data; A generation module is used to generate alarm information of the target business data when the consumption delay is greater than the delay threshold corresponding to the target service, so as to locate the business data with consumption delay.

9. A data monitoring device, characterized in that: A business service device applied to a data processing system, wherein the business service device runs a Flink system, and the data processing system further includes a monitoring device, wherein the device includes: A receiving module, used to receive an execution instruction of a target service; the target service is one of multiple business services; A reading module, used for reading target business data of the target service in the first Kafka message queue through the Flink system; the target business data is one of the multiple business data in the first Kafka message queue consumed by the business service device; A generation module, configured to generate operation data of the target business data, wherein the operation data at least includes: a service identifier of the target service, a consumption time of the target business data, and an event time of the target business data; the consumption time is the moment when the business data of the business service is read by the business service device through the Flink system; A sending module is used to send the operation data to the monitoring device, so that the monitoring device calculates the difference between the consumed time and the event time after acquiring the operation data, and obtains the consumption delay of the target business data, so as to obtain the consumption delay of each business data consumed by the business service device, and determines whether it is necessary to send an alarm message for each business data. When the consumption delay is greater than the delay threshold corresponding to the target service, an alarm message for the target business data is generated to locate the business data where consumption delay occurs.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data monitoring method as described in any one of claims 1 to 4, or the computer program, when executed by the processor, implements the steps of the data monitoring method as described in any one of claims 5 to 7.

Images