Stream processing job detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a stream processing job detection method and device, electronic equipment and a storage medium. The method comprises the steps that a target burying point corresponding to a target task is determined, the target task is a stream processing job for processing and analyzing real-time data in a stream processing framework, the target burying point is a preset program embedded in a target operator of the stream processing framework, and the target operator is used for executing the target task; obtaining target index information of the target task in the running process based on the target burying point, wherein the target index information is used for indicating the running performance of the target task; a detection result of the target task is determined based on the target index information and preset detection logic, the preset detection logic is used for judging whether the target index information meets a corresponding detection threshold value or not, and the detection result of the target task is used for indicating whether the target task is abnormal or not in the running process. According to the scheme, the efficiency, the accuracy and the comprehensiveness of carrying out operation detection on the flow processing operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method, device, electronic device and storage medium for detecting a stream processing job. Background Art

[0002] In big data processing, the reliability and timeliness of data affect the correctness of analysis conclusions, and the detection of real-time data quality is achieved through stream processing jobs (such as Flink tasks executed in the Flink framework) executed in a stream processing framework. Therefore, the normal operation of stream processing jobs is crucial for real-time data processing and analysis. However, there is currently no effective detection method for whether a stream processing job is running normally, especially it is difficult to effectively detect in the face of complex task scenarios. At the same time, the existing detection methods for stream processing jobs cannot detect problems in a timely manner, resulting in the accumulation of problems and affecting the accuracy and timeliness of data processing. Summary of the Invention

[0003] The present invention provides a method, device, electronic device and storage medium for detecting a stream processing job to solve the problem that it is difficult to detect in a timely and effective manner whether an abnormal situation occurs during the operation of a stream processing job in a complex task scenario.

[0004] According to one aspect of the present invention, there is provided a method for detecting a stream processing job, the method comprising:

[0005] Determine a target buried point corresponding to a target task, the target task being a stream processing job for processing and analyzing real-time data in a stream processing framework, the target buried point being a preset program embedded in a target operator of the stream processing framework, and the target operator being used to execute the target task;

[0006] Obtain target metric information during the operation of the target task based on the target buried point, the target metric information being used to indicate the running performance of the target task;

[0007] Determine a detection result of the target task based on the target metric information and a preset detection logic, the preset detection logic being to determine whether the target metric information meets a corresponding detection threshold, and the detection result of the target task being used to indicate whether an abnormal situation occurs during the operation of the target task.

[0008] According to another aspect of the present invention, there is provided a device for detecting a stream processing job, the device comprising:

[0009] A first determination module, configured to determine a target buried point corresponding to a target task, the target task being a stream processing job for processing and analyzing real-time data in a stream processing framework, the target buried point being a preset program embedded in a target operator of the stream processing framework, and the target operator being used to execute the target task;

[0010] A target metric information acquisition module, configured to acquire target metric information during the operation of a target task based on a target data point, where the target metric information is used to indicate the operation performance of the target task;

[0011] A second determination module, configured to determine a detection result of the target task based on the target metric information and a preset detection logic, where the preset detection logic is to determine whether the target metric information meets a corresponding detection threshold, and the detection result of the target task is used to indicate whether an abnormality occurs during the operation of the target task.

[0012] According to another aspect of the present invention, there is provided an electronic device, including:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the stream processing job detection method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the stream processing job detection method according to any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention realizes configuring a target data point in the target operator for executing a target task by determining a target data point corresponding to the target task, where the target task is a stream processing job for processing and analyzing real-time data in a stream processing framework, and the target data point is a preset program embedded in the target operator of the stream processing framework, and the target operator is used to execute the target task; acquiring target metric information during the operation of the target task based on the target data point, where the target metric information is used to indicate the operation performance of the target task; determining a detection result of the target task based on the target metric information and a preset detection logic, where the preset detection logic is to determine whether the target metric information meets a corresponding detection threshold, and the detection result of the target task is used to indicate whether an abnormality occurs during the operation of the target task; realizes acquiring target metric information during the operation of the stream processing job through the target data point configured in the target operator, so as to facilitate comparing the target metric information with the detection threshold adopted by the preset detection logic, and thus determining whether an abnormality occurs during the operation of the stream processing job according to the comparison result between the target metric information and the detection threshold, improves the efficiency, accuracy, and comprehensiveness of the operation detection of the stream processing job, enables the operation detection of the stream processing job to cope with complex task scenarios, and at the same time avoids the accumulation of abnormal situations in the stream processing job from affecting the processing of real-time data.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0020] Figure 1 Flowchart of a method for detecting a stream processing job provided by an embodiment of the present invention;

[0021] Figure 2 Flowchart of another method for detecting a stream processing job provided by an embodiment of the present invention;

[0022] Figure 3 Architecture diagram of a system for detecting a stream processing job provided by an embodiment of the present invention;

[0023] Figure 4 Flowchart of yet another method for detecting a stream processing job provided by an embodiment of the present invention;

[0024] Figure 5 Structural schematic diagram of a device for detecting a stream processing job provided by an embodiment of the present invention;

[0025] Figure 6 Structural schematic diagram of an electronic device for implementing a method for detecting a stream processing job provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description, claims and the above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] It can be understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0029] Figure 1 The figure is a flowchart of a method for detecting a stream processing job provided by an embodiment of the present invention. The embodiments of the present invention are applicable to detecting whether an exception occurs during the operation of a stream processing job that processes and analyzes the data obtained by processing and parsing the logs generated by a financial institution. This method can be executed by a stream processing job detection device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device for implementing the stream processing job detection method. As Figure 1 shown, the method includes:

[0030] S101. Determine a target buried point corresponding to a target task, where the target task is a stream processing job that processes and analyzes real-time data in a stream processing framework, the target buried point is a preset program embedded in a target operator of the stream processing framework, and the target operator is used to execute the target task.

[0031] In the embodiments of the present invention, the stream processing framework may refer to a framework for processing and analyzing real-time data. Among them, the real-time data may come from sensor data, message queue data, log data, databases, etc. Exemplarily, the real-time data may be the data obtained by processing and parsing the logs generated by a financial institution. Processing and analyzing the real-time data may include converting the real-time data into a data stream and performing basic operations such as format conversion, mapping, filtering, and aggregation on the real-time data. Furthermore, the task of processing and analyzing the real-time data executed in the stream processing framework may be used as a stream processing job, such as a Flink task.

[0032] The streaming processing job can be executed by a target operator included in a streaming processing framework. Herein, the target operator can refer to a basic unit for operating on and transforming real-time data in the streaming processing framework. Exemplarily, the target operator can be a basic transformation operator, an aggregation operator, a window operator, etc. The target data tracing can refer to a program pre-written according to the actual needs of the user for obtaining specified data. Specifically, by embedding the pre-written target data tracing into the target operator, when the target operator executes the target task, the expected data can be obtained through the target data tracing.

[0033] As an alternative implementation manner of an embodiment of the present invention, the target operator includes: a data source operator and a data processing operator; the data source operator is used for obtaining real-time data and converting the real-time data into a data stream; the data processing operator is used for performing transformation and analysis operations on the data stream to generate a new data stream.

[0034] In an embodiment of the present invention, the target operator can include a data source (Source) operator and a data processing (Transformation) operator. Herein, the data source operator can refer to an input data source in the streaming processing framework, serving as a starting point for executing the target task, and being used for obtaining real-time data from the outside and introducing the real-time data into the streaming processing framework to convert the real-time data into a data stream. The data processing operator can include operators such as Map, FlatMap, Filter, Aggregate, and Window, and is used for performing operations such as mapping, filtering, flattening, and aggregating on the data stream to convert the data stream into a new data stream.

[0035] S102. Obtain target metric information during the running of the target task based on the target data tracing, where the target metric information is used to indicate the running performance of the target task.

[0036] In an embodiment of the present invention, the running performance of the target task can refer to the running efficiency, response performance, stability, etc. of the target task. When the target indication information indicates that the running performance of the target task does not meet the expected conditions, it can be considered that there is an abnormal situation during the running of the target task. Specifically, when the target operator executes the target task, the target data tracing embedded in the target operator is started. Further, through the target data tracing, the target metric information indicating the running performance of the target task during the running of the target operator executing the target task can be obtained.

[0037] As an alternative implementation of the embodiment of the present invention, the target metric information includes at least one of throughput, latency, out-of-order ratio, and processing time; the throughput is the amount of real-time data processed by the target operator per second; the latency is the time difference between the generation of real-time data and its entry into the target operator, and the out-of-order ratio is the proportion of out-of-order data in the real-time data segments processed by the target operator; the out-of-order data is the data processed by the target operator that is inconsistent with the data sending order; the processing time is the time value used by the target operator to process real-time data.

[0038] In the embodiment of the present invention, the throughput can be used to indicate the processing performance of the target operator for real-time data during the execution of the target task, and to determine whether there is a data stream interruption during the operation of the target task. The latency can be used to indicate the response speed of the target operator for real-time data during the execution of the target task, and to determine whether the target operator meets the real-time requirement. The out-of-order ratio can be used to indicate the amount of out-of-order data that occurs during the execution of the target task by the target operator. The processing time can be used to indicate the efficiency of the target operator in processing real-time data during the execution of the target task.

[0039] S103. Determine the detection result of the target task based on the target metric information and the preset detection logic. The preset detection logic is to judge whether the target metric information meets the corresponding detection threshold, and the detection result of the target task is used to indicate whether an abnormality occurs during the operation of the target task.

[0040] In the embodiment of the present invention, the preset detection logic is preset based on the target metric information. Exemplarily, when the target metric information is throughput, the preset detection logic can be "the throughput is not less than 100 pieces / second"; when the target metric information is latency, the preset detection logic can be "the latency does not exceed 5 seconds"; when the target metric information is out-of-order ratio, the preset detection logic can be "the out-of-order ratio does not exceed 5%"; when the target metric information is processing time, the preset detection logic can be "the processing time does not exceed 10 seconds".

[0041] Whether an abnormality occurs during the operation of the target task may refer to whether there are situations such as data stream interruption, abnormal data, processing latency, or out-of-order data during the operation of the target task. A data stream interruption in the target task may cause data loss or incompleteness; the presence of abnormal data in the target task will affect the accuracy of subsequent data analysis; a relatively large processing latency of the target task will cause the data to not be provided to downstream applications in a timely manner, affecting task decision-making; the existence of out-of-order data in the target task will interfere with the normal data processing logic.

[0042] Specifically, by comparing the target metric information obtained from the target buried point with the preset detection threshold in the preset detection logic, it is determined whether the target metric information meets the corresponding detection threshold according to the comparison result between the target metric information and the detection threshold. Furthermore, if the target metric information meets the corresponding detection threshold, the detection result of the target task indicates that no abnormality occurs during the operation of the target task. If the target metric information does not meet the corresponding detection threshold, the detection result of the target task indicates that an abnormality occurs during the operation of the target task.

[0043] Exemplarily, if the target metric information is throughput and does not meet the corresponding detection threshold, it indicates that a disconnection occurs in the target task; if the target metric information is latency and does not meet the corresponding detection threshold, it indicates that a processing delay occurs in the target task; if the target metric information is out-of-order ratio and does not meet the corresponding detection threshold, it indicates that out-of-order data appears in the target task.

[0044] As an optional implementation manner of the embodiment of the present invention, the stream processing job detection method provided by the embodiment of the present invention further includes: when the detection result indicates that an abnormality occurs during the operation of the target task, generating a task abnormality prompt message and displaying the task abnormality prompt message, where the task abnormality prompt message is used to indicate the abnormality information that occurs in the target task.

[0045] Specifically, when the target metric information obtained from the target buried point does not meet the preset detection threshold in the preset detection logic, it indicates that an abnormality occurs during the operation of the target task. Furthermore, a task abnormality prompt message of the target task can be generated and displayed. The task abnormality prompt message may include the target operator configured with the target buried point and the target metric information obtained from the target buried point, and is used to indicate the target operator corresponding to the occurrence of the abnormality in the target task and the specific situation of the occurrence of the abnormality.

[0046] Exemplarily, the task abnormality prompt message may be a prompt message in at least one form of text, graphics, and audio. Optionally, displaying the task abnormality prompt message may be by playing through a prompt display device and / or sending the task abnormality prompt message to a target prompt device, etc. The prompt display device or the target prompt device may be a digital computer or a mobile device, etc. By displaying the task abnormality prompt message, relevant operation and maintenance personnel can quickly locate the position where the abnormality occurs in the target task based on the task abnormality prompt message, so as to optimize the job logic and improve the operation efficiency and reliability of the target task.

[0047] In the technical solution of the embodiment of the present invention, by determining the target buried point corresponding to the target task, where the target task is a stream processing job for processing and analyzing real-time data in a stream processing framework, and the target buried point is a preset program embedded in the target operator of the stream processing framework, and the target operator is used to execute the target task, it is realized that the target buried point is configured in the target operator that executes the target task; based on the target buried point, the target metric information during the running of the target task is obtained, and the target metric information is used to indicate the running performance of the target task; based on the target metric information and the preset detection logic, the detection result of the target task is determined, and the preset detection logic is to judge whether the target metric information meets the corresponding detection threshold, and the detection result of the target task is used to indicate whether an abnormality occurs during the running of the target task; it is realized that the target metric information during the running of the stream processing job is obtained through the target buried point configured in the target operator, so that the target metric information can be compared with the detection threshold adopted by the preset detection logic, and thus it is determined whether an abnormality occurs during the running of the stream processing job according to the comparison result between the target metric information and the detection threshold, improving the efficiency, accuracy and comprehensiveness of the running detection of the stream processing job, enabling the running detection of the stream processing job to cope with complex task scenarios, and at the same time avoiding the accumulation of abnormal situations in the stream processing job from affecting the processing of real-time data.

[0048] Figure 2 It is a flowchart of another method for detecting a stream processing job provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the target buried point corresponding to the target task in the foregoing embodiment on the basis of the technical solution of the foregoing embodiment. For the solutions not described in detail in this embodiment, see the foregoing embodiment. This embodiment can be combined with each optional solution in the foregoing one or more embodiments. As Figure 2 shown, the method includes:

[0049] S201. Determine the target operator where the target buried point expected to be embedded configured in the middle platform interface, and the middle platform interface is a front-end interface for displaying the target task.

[0050] In the embodiment of the present invention, the middle platform interface may refer to a pre-established front-end interface for configuring and displaying the target task. Specifically, before performing the running detection of the target task, the target operator where the target buried point is expected to be embedded, the target metric information expected to be collected by the target buried point, and the detection threshold that the target metric information in the preset detection logic meets can be pre-configured through the middle platform interface.

[0051] S202. When the target task is started, modify the directed acyclic graph of the target task and embed the target buried point into the corresponding target operator. The directed acyclic graph is used to describe the flow and dependency relationship of real-time data in the target task.

[0052] In an embodiment of the present invention, a directed acyclic graph may refer to a graph structure composed of vertices and directed edges. The vertices in the directed acyclic graph may be target operators, representing processing operations performed on real-time data; the directed edges in the directed acyclic graph may represent the flow direction of real-time data and the dependency relationships between different target operators. Specifically, when a target operator starts to execute a target task, the target buried point can be embedded into the target operator expected in the middle platform interface by automatically modifying the vertices included in the directed acyclic graph corresponding to the target task.

[0053] S203. Obtain target metric information during the running of the target task based on the target buried point, where the target metric information is used to indicate the running performance of the target task.

[0054] S204. Determine the detection result of the target task based on the target metric information and a preset detection logic. The preset detection logic is to judge whether the target metric information meets the corresponding detection threshold, and the detection result of the target task is used to indicate whether an abnormality occurs during the running of the target task.

[0055] As an optional implementation manner of an embodiment of the present invention, obtaining target metric information during the running of the target task based on the target buried point includes: starting the target buried point when the target task is running, and collecting the target metric information in real time through the target buried point; pushing the target metric information to a time series database for storage, and the time series database can store or query the target metric information based on the time dimension order.

[0056] In an embodiment of the present invention, a time series database may refer to a database used to store, manage, and analyze time series data, such as the VictoriaMetrics database. The data stored in the time series database is attached with time tags, and the stored data can be arranged in time order to represent the change of data over time.

[0057] Specifically, during the running of the target task, the target metric information collected in real time through the target buried point embedded in the target operator can be added with time tags. Furthermore, based on the address corresponding to the pre-configured time series database, the target metric information added with time tags can be pushed to the time series database for storage.

[0058] Optionally, based on the address of the pre-configured message middleware, the target metric information added with time tags is pushed to the message middleware, and then the target metric information added with time tags is pushed to the time series database for storage through the message middleware, so as to filter or verify the collected data through the message middleware and improve the accuracy of the stored data.

[0059] As an alternative implementation of the embodiment of the present invention, determining the detection result of the target task based on the target metric information and the preset detection logic includes: querying the target metric information within a preset time window stored in the time series database based on a preset time interval; determining the detection result based on the comparison result between the target metric information within the preset time window and the detection threshold used in the preset detection logic.

[0060] Specifically, a scheduled task can be configured to periodically query the time series database at a preset time interval, and based on the time tags added to the target metric information stored in the time series database, obtain the target metric information within a preset time window. Among them, the preset time window can refer to a configurable time range that can be updated in real time. Furthermore, by comparing the target metric information within the preset time window with the detection threshold used in the preset detection logic, the comparison result between the target metric information within the preset time window and the detection threshold can be obtained, thereby determining the detection result of the target task.

[0061] Optionally, determine the reference metric information based on the target metric information within the preset time window, and then determine the detection result of the target task based on the comparison result between the reference metric information and the detection threshold used in the preset detection logic. Among them, the reference metric information can refer to the maximum value, minimum value, or average value of the target metric information within the preset time window.

[0062] Exemplarily, refer to Figure 3 the architecture diagram of the stream processing job detection system shown. Taking the real-time Flink task as an example for the target task and the VictoriaMetrics database as an example for the time series database. Figure 3 In the middle platform interface, the target operator, target metric information, and detection threshold expected to be embedded with target data points can be configured, and the scheduled task and the target task can be started. Among them, the scheduled task is used to periodically query the target metric information within a preset time window stored in the time series database. Furthermore, a series of task exception prompt messages can be included in the alarm list generated according to the preset detection logic.

[0063] Exemplarily, Figure 4 is the flowchart of another stream processing job detection method provided by the embodiment of the present invention. As shown in Figure 4As shown, the target task takes a real-time Flink task as an example, and the time series database takes the VictoriaMetrics database as an example. Users can define the buried point location, metric type, and detection threshold in the middleware. Among them, the buried point location can be the target operator where the target buried point is expected to be embedded, and the metric type can be the target metric information that the target buried point is expected to collect. Furthermore, start the target task, embed the target buried point into the target operator, and collect the target metric information through the target buried point and push the target metric information to the time series database for storage. At the same time, start a scheduled task to query the target metric information stored in the time series database within a preset time window regularly, and calculate and determine the maximum value, minimum value, or average value of the target metric information within the preset time window. Then, generate alarm data based on the preset detection logic and display it through the middleware. Among them, the alarm data can be task exception prompt information. Finally, the operation and maintenance personnel can optimize the abnormal situation of the target task based on the task exception prompt information.

[0064] In the technical solution of the embodiment of the present invention, by determining the target operator where the target buried point configured on the middleware interface is expected to be embedded, the middleware interface is the front-end interface for displaying the target task; when the target task is started, modify the directed acyclic graph of the target task, and embed the target buried point into the corresponding target operator. The directed acyclic graph is used to describe the flow and dependency relationship of real-time data in the target task, which realizes pre-configuring the target operator where the target buried point is expected to be embedded through the middleware interface, and automatically modifying the directed acyclic graph of the target task when the target task is started to embed the target buried point into the target operator, improving the flexibility and timeliness of target buried point configuration; obtaining the target metric information of the target task during operation based on the target buried point, and the target metric information is used to indicate the operation performance of the target task; determining the detection result of the target task based on the target metric information and the preset detection logic, and the preset detection logic is to judge whether the target metric information meets the corresponding detection threshold. The detection result of the target task is used to indicate whether an abnormality occurs during the operation of the target task, improving the efficiency, accuracy, and comprehensiveness of the operation detection of the stream processing job, enabling the operation detection of the stream processing job to handle complex task scenarios, and at the same time avoiding the accumulation of abnormal situations in the stream processing job from affecting the processing of real-time data.

[0065] Figure 5 It is a schematic structural diagram of a stream processing job detection device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to detecting whether an abnormality occurs during the operation of a stream processing job that processes and analyzes data obtained from the processing and parsing of logs generated by financial institutions. This device can be implemented in the form of hardware and / or software. As Figure 5 shown, the device includes: a first determination module 301, a target metric information acquisition module 302, and a second determination module 303. Among them,

[0066] A first determination module 301, configured to determine a target data point for a target task, where the target task is a stream processing job for processing and analyzing real-time data in a stream processing framework, the target data point is a preset program embedded in a target operator of the stream processing framework, and the target operator is used to execute the target task;

[0067] A target metric information acquisition module 302, configured to acquire target metric information during the running of the target task based on the target data point, where the target metric information is used to indicate the running performance of the target task;

[0068] A second determination module 303, configured to determine a detection result of the target task based on the target metric information and a preset detection logic, where the preset detection logic is to determine whether the target metric information meets a corresponding detection threshold, and the detection result of the target task is used to indicate whether an abnormality occurs during the running of the target task.

[0069] Based on any of the above optional technical solutions, optionally, the target operator includes: a data source operator and a data processing operator; the data source operator is used to acquire real-time data and convert the real-time data into a data stream; the data processing operator is used to perform conversion and analysis operations on the data stream to generate a new data stream.

[0070] Based on any of the above optional technical solutions, optionally, the first determination module 301 includes: a target operator determination unit and a target data point embedding unit. Among them, the target operator determination unit is configured to determine the target operator in which the target data point expected to be embedded configured in the middle platform interface, and the middle platform interface is a front-end interface for displaying the target task; the target data point embedding unit is configured to modify the directed acyclic graph of the target task when the target task is started, and embed the target data point into the corresponding target operator, and the directed acyclic graph is used to describe the flow and dependency relationship of real-time data in the target task.

[0071] Based on any of the above optional technical solutions, optionally, the target metric information includes at least one of throughput, latency, out-of-order ratio, and processing time; the throughput is the amount of real-time data processed by the target operator per second; the latency is the time difference between the generation of real-time data and the entry into the target operator; the out-of-order ratio is the proportion of out-of-order data in the real-time data segment processed by the target operator; the out-of-order data is the data whose processing order by the target operator is inconsistent with the data sending order; the processing time is the time value used by the target operator to process real-time data.

[0072] Based on any of the above optional technical solutions, optionally, the target metric information acquisition module 302 includes: a target instrumentation startup unit and a target metric information storage unit. Among them, the target instrumentation startup unit is used to start target instrumentation when the target task is running, and collect target metric information in real time through the target instrumentation; the target metric information storage unit is used to push the target metric information to the time series database for storage, and the time series database can store or query the target metric information based on the time dimension order.

[0073] Based on any of the above optional technical solutions, optionally, the second determination module 303 includes: a target metric information query unit and a detection result determination unit. Among them, the target metric information query unit is used to query the target metric information within a preset time window stored in the time series database based on a preset time interval; the detection result determination unit is used to determine the detection result based on the comparison result between the target metric information within the preset time window and the detection threshold used by the preset detection logic.

[0074] Based on any of the above optional technical solutions, optionally, the stream processing job detection device of the embodiment of the present invention further includes: an exception prompt information generation module. Among them, the exception prompt information generation module is used to generate a task exception prompt information and display the task exception prompt information when the detection result indicates that an exception occurs during the running of the target task, and the task exception prompt information is used to indicate the exception information that appears in the target task.

[0075] In the technical solution of the embodiment of the present invention, the first determination module 301 determines a target data point corresponding to a target task. The target task is a stream processing job for processing and analyzing real-time data in a stream processing framework. The target data point is a preset program embedded in a target operator of the stream processing framework. The target operator is used to execute the target task, thereby realizing the configuration of the target data point in the target operator for executing the target task. The target metric information acquisition module 302 acquires target metric information during the operation of the target task based on the target data point. The target metric information is used to indicate the running performance of the target task. The second determination module 303 determines the detection result of the target task based on the target metric information and a preset detection logic. The preset detection logic is to determine whether the target metric information meets the corresponding detection threshold. The detection result of the target task is used to indicate whether an abnormality occurs during the operation of the target task. It realizes the acquisition of target metric information during the operation of the stream processing job through the target data point configured in the target operator, so as to facilitate the comparison of the target metric information with the detection threshold used in the preset detection logic, and thus determine whether an abnormality occurs during the operation of the stream processing job according to the comparison result between the target metric information and the detection threshold, improving the efficiency, accuracy, and comprehensiveness of the running detection of the stream processing job, enabling the running detection of the stream processing job to handle complex task scenarios, and at the same time avoiding the accumulation of abnormal situations in the stream processing job from affecting the processing of real-time data.

[0076] The stream processing job detection device provided by the embodiment of the present invention can execute the stream processing job detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0077] Figure 6 It is a schematic structural diagram of an electronic device for implementing a stream processing job detection method provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0078] As Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0079] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0080] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the stream processing job detection method.

[0081] In some embodiments, the stream processing job detection method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the stream processing job detection method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the stream processing job detection method by any other appropriate means (e.g., by means of firmware).

[0082] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0083] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0084] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0086] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0087] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0088] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0089] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting a stream processing job, characterized in that: The method comprises: Determine a target tracking point corresponding to a target task, wherein the target task is a stream processing job for processing and analyzing real-time data in a stream processing framework, and the target tracking point is a preset program embedded in a target operator of the stream processing framework, and the target operator is used to execute the target task; Acquire target indicator information of the target task during operation based on the target tracking point, where the target indicator information is used to indicate the operation performance of the target task; The detection result of the target task is determined based on the target indicator information and the preset detection logic. The preset detection logic is used to determine whether the target indicator information meets the corresponding detection threshold. The detection result of the target task is used to indicate whether an abnormality occurs during the operation of the target task.

2. The method according to claim 1, characterized in that: The target operator includes: a data source operator and a data processing operator; the data source operator is used to obtain real-time data and convert the real-time data into a data stream; the data processing operator is used to perform conversion and analysis operations on the data stream to generate a new data stream.

3. The method according to claim 1, characterized in that Determine the target tracking points corresponding to the target task, including: Determine the target operator that the target embedding point configured in the middle platform interface is expected to be embedded in, and the middle platform interface is a front-end interface for displaying the target task; When the target task is started, the directed acyclic graph of the target task is modified, and the target embedding point is embedded into the corresponding target operator. The directed acyclic graph is used to describe the flow and dependency of real-time data in the target task.

4. The method according to claim 1, characterized in that: The target indicator information includes: at least one of throughput, latency, out-of-order ratio and processing time; the throughput is the amount of real-time data processed by the target operator per second; the latency is the time difference between the generation of real-time data and the entry into the target operator; the out-of-order ratio is the proportion of out-of-order data in the real-time data segment processed by the target operator; the out-of-order data is the data processed by the target operator that is inconsistent with the order in which the data is sent; the processing time is the time value used by the target operator to process real-time data.

5. The method according to claim 1, characterized in that Acquiring target indicator information of the target task during operation based on the target tracking point includes: When the target task is running, the target tracking point is started, and the target indicator information is collected in real time through the target tracking point; The target indicator information is pushed to a time series database for storage, and the time series database can store or query the target indicator information based on the time dimension sequence.

6. The method according to claim 5, characterized in that Determining the detection result of the target task based on the target indicator information and the preset detection logic includes: Querying the target indicator information within a preset time window stored in the time series database based on a preset time interval; The detection result is determined based on a comparison result between the target indicator information within a preset time window and a detection threshold adopted by the preset detection logic.

7. The method according to claim 1, characterized in that The method further comprises: When the detection result indicates that an abnormality occurs in the target task during operation, task abnormality prompt information is generated and displayed, where the task abnormality prompt information is used to indicate abnormal information occurring in the target task.

8. A stream processing job detection device, characterized in that: The device comprises: A first determination module is used to determine a target embedding point corresponding to a target task, wherein the target task is a stream processing job for processing and analyzing real-time data in a stream processing framework, and the target embedding point is a preset program embedded in a target operator of the stream processing framework, and the target operator is used to execute the target task; A target indicator information acquisition module, used to acquire the target indicator information of the target task during operation based on the target tracking point, wherein the target indicator information is used to indicate the operation performance of the target task; The second determination module is used to determine the detection result of the target task based on the target indicator information and the preset detection logic. The preset detection logic is used to determine whether the target indicator information meets the corresponding detection threshold. The detection result of the target task is used to indicate whether the target task has an abnormality during operation.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the stream processing job detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the stream processing job detection method according to any one of claims 1 to 7 when executed.

Images