Job monitoring method and device, system, equipment, computer readable storage medium

Abstract

The present disclosure provides a kind of job monitoring method and device, system, equipment, computer readable storage medium, the method comprises: the monitoring index data of received streaming job is transmitted to the data interface of at least one data reporting link preconfigured;Different data reporting links correspond to different monitoring service types;The monitoring index data of streaming job of each data reporting link is executed the monitoring processing corresponding to monitoring service type, and the corresponding monitoring processing result is obtained.According to the embodiment of the present disclosure, the monitoring processing efficiency can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a job monitoring method, apparatus, system, device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of Big Data technology, real-time computing is finding increasingly widespread applications in the field. "Real-time is the future," and as real-time technology matures, real-time computing will play an increasingly important role in the Big Data field. For example, through real-time computation of streaming data, large-scale flowing data from different data sources can be acquired in real time, and valuable information can be obtained through real-time computation and analysis. To improve the operational stability of streaming jobs in real-time computing, it is necessary to monitor the job's operational status during the process. Summary of the Invention

[0003] This disclosure provides a method, apparatus, system, device, and computer-readable storage medium for monitoring operations, which can improve the efficiency of monitoring and processing of monitoring index data for streaming operations.

[0004] In a first aspect, this disclosure provides a job monitoring method, which includes: transmitting received monitoring index data of streaming jobs to the data interface of at least one pre-configured data reporting link; wherein different data reporting links correspond to different monitoring service types; and performing monitoring processing corresponding to the monitoring service type on the monitoring index data of streaming jobs for each data reporting link to obtain the corresponding monitoring processing result.

[0005] Secondly, this disclosure provides a job monitoring device, which includes: a transmission module for transmitting received monitoring index data of streaming jobs to the data interface of at least one pre-configured data reporting link; wherein different data reporting links correspond to different monitoring service types; and a processing module for performing monitoring processing corresponding to the monitoring service type on the monitoring index data of streaming jobs of each data reporting link to obtain the corresponding monitoring processing result.

[0006] Thirdly, this disclosure provides a monitoring architecture system, which includes: a real-time data computing platform, a resource management system, and a job monitoring system; the real-time data computing platform is used to submit streaming jobs to the resource management system; the resource management system is used to run streaming jobs and send the monitoring indicator data generated by the running streaming jobs to the job monitoring system; the job monitoring system is used to receive the monitoring indicator data of streaming jobs and execute the aforementioned job monitoring method.

[0007] Fourthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the above-described job monitoring method.

[0008] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described job monitoring method when executed by a processor / processor core.

[0009] The job monitoring method in the embodiments provided in this disclosure can provide different data reporting links for the monitoring index data of streaming jobs. Since different data reporting links can correspond to different monitoring service types, the job monitoring method of this disclosure can perform monitoring processing of different monitoring service types on the monitoring index data of streaming jobs. This can provide different monitoring service types for monitoring the job operation status of streaming jobs, realize the diversification of service types for monitoring the job operation status, and improve the utilization rate of the monitoring index data of streaming jobs. Thus, through the efficient utilization of the monitoring index data, the monitoring processing efficiency is improved. Furthermore, according to this job monitoring method, when system errors corresponding to different monitoring service types occur, it is beneficial to quickly locate the problematic data reporting link and improve the efficiency of problem investigation.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0012] Figure 1 This is a schematic diagram of the architecture of a job monitoring system in related technologies;

[0013] Figure 2 This is a schematic diagram of a scenario for a job monitoring system provided in an embodiment of this disclosure;

[0014] Figure 3 A flowchart of the job monitoring method provided in this embodiment of the disclosure;

[0015] Figure 4A schematic diagram of a job monitoring page provided in an embodiment of this disclosure;

[0016] Figure 5 This is a schematic diagram of the monitoring architecture system provided in the embodiments of this disclosure;

[0017] Figure 6 A schematic diagram of the page before redirection in the data visualization platform provided in this embodiment of the disclosure;

[0018] Figure 7 This is a schematic diagram of the structure of the job monitoring system provided in the embodiments of this disclosure;

[0019] Figure 8 A flowchart of the job monitoring method provided in this embodiment of the disclosure;

[0020] Figure 9 A block diagram of a work monitoring device provided in an embodiment of this disclosure;

[0021] Figure 10 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0024] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0026] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0027] With the development of internet technology and the continuous advancement of applications such as real-time search, advertising recommendations, social networks, and online log analysis, streaming data is emerging as a new data format. Streaming data can be understood as a large, fast, and uninterrupted sequence of events. In different scenarios, streaming data can take various forms, such as real-time queries, user clicks, online logs, and streaming media.

[0028] Streaming computing refers to computation on streaming data. In streaming computing, data sources are input, output, and processed in the form of data streams, exhibiting strong sequentiality and continuous arrival characteristics. Real-time streaming computing refers to performing real-time computation on streaming data. Real-time computing emphasizes real-time processing, requiring low latency from data generation to its impact on the final result. Through real-time streaming computing, large-scale flowing data from different data sources can be acquired in real time, and valuable information can be obtained through real-time computation and analysis.

[0029] Streaming computing frameworks are distributed real-time processing frameworks that can process streaming jobs by running continuously. A streaming job is a computational job that transforms data into a stream and performs filtering, transformation, and other operations on the stream. Streaming computing frameworks typically use open-source distributed systems, which have the advantages of low latency, scalability, and fault tolerance, ensuring the real-time performance and high availability of streaming computing.

[0030] In some scenarios, real-time computing platforms are used to manage and schedule streaming jobs running within streaming computing frameworks. To improve the operational stability of streaming jobs, it is also necessary to monitor their internal states during execution to facilitate timely and rapid handling of abnormal streaming jobs and ensure business stability. However, current technologies for monitoring streaming job metrics suffer from low resource utilization and low monitoring efficiency. Specifically, this will be discussed in the following... Figure 1 Please provide a detailed explanation.

[0031] Figure 1 A schematic diagram of the architecture of a job monitoring system in related technologies is shown. Figure 1 The system includes a resource management system 10, a transfer device 20, a data acquisition device 30, and a visualization platform 40; the resource management system 10 runs multiple streaming jobs, including but not limited to... Figure 1The following are examples of flow jobs 11, 12 and 13.

[0032] exist Figure 1 In the process, each streaming job, when running in the resource management system 10, can push its own streaming job monitoring indicator reporting component to the relay device 20 based on the job monitoring indicator reporting component supported by the streaming computing framework itself. The relay device 20 can be used to push the received monitoring indicators to the acquisition device 30. The acquisition device 30 can be used to query and process the monitoring indicator data. The visualization platform 40 can be used to visualize the monitoring indicator data processed by the information data acquisition device 30.

[0033] pass Figure 1 It can be seen that, on the one hand, this job monitoring system mainly monitors the running status of streaming computing jobs by providing a visualization function of query results, but it cannot perform other types of monitoring processing on the monitoring indicator data of streaming computing jobs, resulting in low resource utilization and monitoring processing efficiency of the monitoring indicator data; on the other hand, when there are a large number of streaming jobs that need to be monitored, this job monitoring system may affect the performance of the entire job monitoring system. Taking the relay device 20 as an information collection relay machine (pushgateway) and the acquisition device 30 as an information data acquisition server (Prometheus server) developed in Go language as an example, the Prometheus server can pull the monitoring indicator data of streaming jobs from the pushgateway and store and query the received monitoring indicator data of streaming jobs. Since the pushgateway does not support periodic data cleanup (e.g., periodic cleanup of monitoring indicator data that achieves TTL by limiting the time to live (TTL) of data), the system is flawed. When the pushgateway needs to monitor the monitoring metrics of multiple streaming jobs, a performance bottleneck may occur, which will affect the performance of the entire job monitoring system and result in low processing efficiency.

[0034] According to one or more embodiments of the present disclosure, the job monitoring method can provide different monitoring service types for streaming jobs to monitor job operation status, improve the resource utilization and monitoring processing efficiency of monitoring indicator data of streaming jobs; it can perform high-performance querying and storage of monitoring indicator data through a time-series database; and the monitoring architecture system of the present disclosure includes a real-time management platform, on which a cache space can be opened to cache the set of indicator information that needs to be reported, and by setting the effective duration of information storage in the cache space, the cached information can be cleaned up periodically to improve the system performance of the monitoring architecture system.

[0035] The following describes in detail, with reference to the accompanying drawings, the operation monitoring method, apparatus, system, device, and computer-readable storage medium of the present disclosure. Figure 2 A schematic diagram illustrating a scenario of a job monitoring system according to an embodiment of this disclosure is shown. Figure 2 In this scenario, the real-time computing platform 210, the resource management system 220, and the job monitoring system 230 are included.

[0036] like Figure 2 As shown, the real-time computing platform 210 can submit the generated streaming jobs to the resource management system 220; the resource management system 220 is used to provide resource management and task scheduling for streaming jobs based on the streaming computing framework, thereby running the streaming jobs; the job monitoring system 230 can be used to receive the monitoring index data of the streaming jobs reported by the resource management system 220, and execute the job monitoring method of this embodiment on the received monitoring index data to obtain the corresponding monitoring processing results. Figure 2 Resource Management System 220 and Figure 1 The resource management system 10 in the system can be the same or different resource management systems.

[0037] For example, the resource management system 220 may include, but is not limited to, a distributed resource management system (YetAnother Resource Negotiator, YARN) or a distributed resource management system (Mesos).

[0038] For example, streaming computing frameworks may include, but are not limited to, Storm, Spark Streaming, and Flink. Specifically, Storm is a distributed computing framework for streaming event processing; Spark Streaming is a computing framework for splitting continuous real-time input data streams into discrete data streams for streaming processing; and Flink is a stateful big data streaming computing framework.

[0039] For the sake of simplicity, the following embodiments use monitoring metrics reported by streaming jobs based on the Flink framework running in YARN (hereinafter referred to as Flink jobs) as examples to illustrate the specific processing steps of the job monitoring method of this disclosure. However, this description should not be construed as limiting the scope or implementation possibilities of this solution. The processing method for streaming jobs generated by running streaming jobs in other resource management systems other than YARN and / or other streaming computing frameworks other than Flink is consistent with the processing method for Flink jobs running in YARN.

[0040] Figure 3 A flowchart illustrating a job monitoring method according to an embodiment of this disclosure is provided. Figure 3In this context, the job monitoring method may include the following steps.

[0041] S310, transmits the received monitoring index data of the streaming job to the data interface of at least one pre-configured data reporting link; wherein, different data reporting links correspond to different monitoring service types.

[0042] In some embodiments, depending on the various data formats of streaming data in different scenarios, streaming jobs may also involve different scenarios. For example, streaming jobs include, but are not limited to, streaming computations on data such as real-time query data, user click data, online log data, and streaming media.

[0043] In some embodiments, the monitoring metrics data for streaming jobs are the monitoring metrics of the streaming job during its execution, such as, but not limited to, the monitoring metrics of Flink jobs during their execution. Monitoring metrics data for streaming jobs can help developers or users better understand the status of the streaming job or the streaming job cluster.

[0044] In this embodiment, the data interface of the data reporting link is used to receive monitoring indicator data of the streaming job. Different data reporting links have different data interfaces, different data transmission flows, and different monitoring service types. The monitoring service type determines the specific execution steps for subsequent monitoring and processing of the monitoring indicator data.

[0045] S320 performs monitoring processing corresponding to the monitoring service type on the monitoring metric data of each data reporting link's streaming job, and obtains the corresponding monitoring processing results.

[0046] In this embodiment of the disclosure, at least one data interface of a data reporting link can be pre-configured for the received monitoring index data of the streaming job. Since different data reporting links correspond to different monitoring service types, transmitting the monitoring index data of the streaming job to the data interface of any pre-configured data reporting link means that a monitoring service type corresponding to any data reporting link has been selected for the monitoring index data of the streaming job. Thus, the monitoring processing corresponding to the monitoring service type can be performed on the received monitoring index data of the streaming job to obtain the corresponding monitoring processing result and realize the monitoring service of the corresponding monitoring server type.

[0047] According to the job monitoring method of this disclosure, compared with the related technologies where the monitoring and processing of streaming jobs can only achieve the visualization of query results and the monitoring service type is single, the utilization rate of the monitoring indicator data of streaming jobs is low. In this disclosure, different data reporting links are provided for the monitoring indicator data of streaming jobs. Since different data reporting links can correspond to different monitoring service types, the job monitoring method of this disclosure can perform monitoring processing corresponding to different monitoring service types on the monitoring indicator data of streaming jobs. This can provide different monitoring service types for the job operation status monitoring of streaming jobs, realize the diversification of the service types for job operation status monitoring, and help improve the resource utilization rate of the monitoring indicator data of streaming jobs. Thus, by making efficient use of the monitoring indicator data, the monitoring processing efficiency is improved. Furthermore, when system errors corresponding to different monitoring service types occur, it is helpful to quickly locate the problematic data reporting link and improve the problem troubleshooting efficiency.

[0048] In some embodiments, in step S110, different data reporting links include a first reporting link and a second reporting link; different monitoring service types include: a monitoring alarm service corresponding to the first reporting link and a monitoring indicator visualization service corresponding to the second reporting link. In this embodiment, step S310 may specifically include: transmitting the received monitoring indicator data to the data interface of the first reporting link and the data interface of the second reporting link to provide the corresponding monitoring alarm service and the corresponding monitoring indicator visualization service.

[0049] In this embodiment, the different data reporting links are the first reporting link and the second reporting link, and the different data reporting links correspond to different monitoring service types. The monitoring service type corresponding to the first reporting link is the monitoring alarm service, and the monitoring service type corresponding to the second reporting link is the monitoring indicator visualization service. Based on this, the job monitoring method of this embodiment can provide monitoring services including alarm services and monitoring indicator visualization services for the monitoring indicator data of streaming jobs.

[0050] It should be understood that the description of the first and second reporting links is intended to distinguish the two reporting links as different reporting links, and does not require or imply any actual relationship or order between the reporting order of the monitoring indicator data involved in these reporting links and the monitoring processing operations.

[0051] In this embodiment of the disclosure, the monitoring and alarm service corresponds to the first reporting link. The monitoring and alarm service can be implemented based on a distributed message distribution system. The distributed message distribution system can build a central control system for message distribution among the various systems involved in the job monitoring and alarm system of this embodiment of the disclosure. After the monitoring index data of the streaming job is reported to the distributed message distribution system, the corresponding monitoring index data can be consumed by the processing component of the streaming job to perform relevant processing on the streaming job for monitoring and alarm purposes.

[0052] In some embodiments, the distributed message distribution system may include, but is not limited to, the distributed publish-subscribe messaging system Kafka, the distributed file system (Hadoop Distributed File System, Hadoop), and the distributed column-oriented open-source database system (HBase), depending on the supported distribution scenarios. The specific system can be selected based on the actual application scenario and monitoring and processing requirements.

[0053] In this embodiment of the disclosure, the monitoring indicator visualization service corresponds to the second reporting link. The monitoring indicator visualization service can be implemented based on a time-series database. A time-series database is a database used to store time-series data. Time-series data refers to data carrying time labels (changing in chronological order, i.e., time-series data).

[0054] In some embodiments, the time-series database may include, but is not limited to, the following, time-series databases, depending on their emphasis on supported functions: InfluxDB, TimescaleDB, and KairosDB. InfluxDB can be used for high-performance querying and storage of time-series data; TimescaleDB supports open-source code of the complete Structured Query Language (SQL); and KairosDB supports data writing using multiple protocols. The specific database to be selected depends on the actual application scenario and monitoring processing requirements.

[0055] In some embodiments, the data interface of the first reporting link is a data interface corresponding to the distributed publish-subscribe messaging system; the data interface of the second reporting link is a data interface corresponding to the time-series database.

[0056] In this embodiment, the distributed publish-subscribe messaging system can be a Kafka system, and the time-series database can be InfluxDB. For the Kafka system, monitoring metric data from streaming jobs is written to the corresponding data interface of the Kafka system, and the processing component of the streaming job consumes the Kafka data, i.e., processes the monitoring metric data published to the Kafka system, and provides an alarm service for the monitoring metric data. For InfluxDB, monitoring metric data from streaming jobs is written to InfluxDB, and InfluxDB is used for high-performance querying and storage of the monitoring metric data, as well as for visualization services of the monitoring metrics based on the monitoring metric data stored in InfluxDB.

[0057] In this embodiment, monitoring metrics data for Flink jobs can be simultaneously reported to both the Kafka system and the InfluxDB database, enabling multi-path reporting of monitoring metrics data for Flink jobs. One path reports the monitoring metrics data of Flink jobs to the Kafka system to provide monitoring and alerting services for streaming jobs; another path reports the monitoring metrics data of Flink jobs to InfluxDB to provide visualization services for subsequent monitoring metrics through distributed storage.

[0058] The processing of monitoring indicator data transmitted to the two data reporting links is described in detail below with reference to specific embodiments. The following embodiments describe specific implementation methods for providing monitoring and alarm services for streaming jobs using monitoring indicator data. In the following embodiments, the monitoring processing corresponding to the monitoring and alarm service is performed on the monitoring indicator data, which can be implemented by the alarm processing component of the streaming job. The alarm processing component can be implemented by a computer program, code, or processing chip.

[0059] In some embodiments, the at least one data reporting link includes a first reporting link corresponding to the monitoring and alarm service; step S120 may specifically include the following steps.

[0060] S11, take the monitoring indicator data of the streaming job of the first reporting link as the monitoring indicator data to be matched.

[0061] S12, Match the monitoring indicator data to be matched with the pre-read alarm rule set; wherein, the alarm rule set includes at least one alarm rule, and each alarm rule corresponds to the job name of each streaming job.

[0062] S13. If the monitoring indicator data to be matched matches the target alarm rule from the alarm rule set, then an alarm message is generated according to the target alarm rule, and the alarm message is used as the corresponding monitoring processing result.

[0063] In this embodiment, when the number of rules in the alarm rule set is multiple (greater than 1), matching of the monitoring indicator data to be matched with multiple alarm rules can be achieved. In related technologies, alarm rule matching is usually performed by matching a single data stream with a single rule, where a single data stream refers to the monitoring indicator data of a single streaming job. However, this embodiment can improve the matching rule, thereby enabling model matching of multiple rules within a single data stream (the monitoring indicator data of a streaming job), improving rule matching efficiency, and thus improving the alarm processing efficiency of monitoring indicator data.

[0064] Taking Flink jobs as an example, the native complex event processing component based on Flink can be denoted as Flink-CEP, where CEP stands for Complex Event Processing. The native Flink-CEP can implement the matching function between monitoring data metrics and alarm rules based on computer programs or code functions. For example, the native Flink-CEP can be the native Flink complex event processing model. Related technologies use the native complex event model to match monitoring metric data and alarm rules for streaming jobs.

[0065] In this example, after the monitoring metric data is transmitted to the Kafka system in the related technology, the native Flink-CEP only supports single stream data and single rule matching. However, after improving the processing method of the native Flink-CEP, the embodiments of this disclosure can achieve the matching of multiple alarm rules for a data stream (monitoring metric data of a streaming job). This rule matching method can improve the rule matching efficiency, thereby improving the alarm processing efficiency of monitoring metric data.

[0066] In some embodiments, the monitoring metric data to be matched corresponds to at least two streaming jobs of the first reporting link; the step of matching the monitoring metric data to be matched with the pre-read alarm rule set in step S12 above may specifically include the following steps.

[0067] S21, Get the job name of the monitoring metric data to be matched.

[0068] In this step, each monitoring metric data has a job name (JobName), which is the name of the streaming job. When the monitoring metric data of the streaming job is received from the resource management system, the job name can be obtained from the attribute information of the monitoring metric data; however, in some scenarios, if the obtained job name is empty, it means that the resource management system did not report the job name when reporting the monitoring metric data of the streaming job; in this case, this embodiment of the disclosure supports directly reading the job name from the running environment information of the streaming job.

[0069] For example, when a resource management system runs a streaming job, it can monitor the job progress through an application manager. The application manager can decide how to run the tasks that make up the entire job. When the amount of data to be processed by the streaming job is small (e.g., less than a predetermined data amount threshold), the application manager will choose to run the streaming job task in its own virtual machine (Java Virtual Machine, JVM) client. When the streaming job runs on the JVM client of the application manager, the attribute information of the monitoring alarm data generated includes the job name.

[0070] Based on this, if the job name value in the attribute information of the monitoring metrics data of streaming jobs reported by the resource management system is empty, the job name can be obtained from the JVM client through the parameter acquisition interface provided by the resource management system.

[0071] S22, group the monitoring indicator data to be matched based on the job name.

[0072] In this step, the monitoring indicator data to be matched after being grouped have the same job name within the same group.

[0073] S23. Based on the job name of the monitoring indicator data to be matched for each group, obtain the corresponding alarm rule from the alarm rule set to obtain the alarm rule subset corresponding to each group.

[0074] In this step, since each rule in the alarm rule set corresponds to a job name, a subset of alarm rules can be filtered out from the alarm rules based on the job name of the monitoring indicator data of each group. This subset can be specifically used to match the monitoring indicator data to be matched within that group, thereby narrowing the search range when matching the monitoring indicator data with the alarm rules.

[0075] S24, match the monitoring indicator data to be matched for each streaming job in the first reporting link with the alarm rule subset corresponding to the group to which it belongs, so as to obtain the target alarm rule that matches the monitoring indicator data to be matched for each streaming job in the first reporting link.

[0076] In this embodiment, monitoring indicator data can be grouped, and the job names within each group can be used to match the corresponding alarm rules in the alarm rule set. Specifically, the monitoring indicator data to be matched for streaming jobs are grouped according to job names. When the grouped monitoring indicator data needs to be matched with alarm rules in subsequent processes, the monitoring indicator data to be matched for each streaming job can be matched with its corresponding alarm rule according to the job name. This allows for centralized processing of monitoring indicator data in the same group and narrows the search range when matching monitoring indicator data with alarm rules, thereby shortening the matching time and improving matching efficiency.

[0077] In some embodiments, each alarm rule in the same subset of alarm rules corresponds to the same preset key value, and the same preset key value is used to identify the same monitoring indicator data to be matched; the above step S23 may specifically include the following steps.

[0078] S31, obtain the key value of the monitoring indicator data to be matched for each streaming job in the first reporting link.

[0079] S32, take the subset of alarm rules corresponding to the key values ​​of the monitoring indicator data to be matched as the subset of alarm rules for each group.

[0080] In this embodiment, a subset of alarm rules corresponding to the key value of the monitoring indicator data to be matched can be obtained, thereby matching the monitoring indicator data to be matched with the rules in the subset of alarm rules.

[0081] As can be seen from the related technologies described in the above embodiments, when matching alarm rules, the native complex event processing component based on Flink uses a processing method of matching a single data stream with a single rule. In other words, the data stream to be matched with a single rule is pre-set, so there is no need to determine the directional information of which data stream the alarm rule is used to match.

[0082] In this embodiment of the disclosure, if the monitoring indicator data to be matched for each streaming job is regarded as a data stream to be matched, then there can be multiple data streams to be matched that need to be matched with the alarm rule subsets corresponding to their respective groups. For example, assuming that the alarm rule set A includes multiple alarm rule subsets, two of which are alarm rule subsets A1 and A2; before matching the alarm rules of the data stream 1 to be matched with the data stream 2 to be matched, the correspondence between the alarm rule subsets and the data streams should be determined in advance, otherwise it will be impossible to determine which alarm rule subset the data stream to be matched should be matched with. After determining the correspondence between the data stream to be matched and the alarm rule subsets, the data stream to be matched can be matched with one or more alarm rules in the corresponding alarm rule subset.

[0083] Therefore, in order to determine the correspondence between the subset of alarm rules and the data stream, the native complex event processing component of Flink can be improved by using the key value of the monitoring metric data to be matched to represent the monitoring metric data, and establishing the correspondence between the alarm rule and the key value. This will give us the correspondence between the alarm rule and the monitoring metric data to be matched. Based on this correspondence, the data stream to be matched is matched with the alarm rules in the corresponding subset of alarm rules.

[0084] In some embodiments, before matching the monitoring indicator data to be matched with the pre-read alarm rule set in step S12, the job monitoring method may further include the following steps.

[0085] S41, read alarm rules from the specified system database at predetermined rule reading intervals.

[0086] In this step, the rule reading interval refers to the time interval between two consecutive reads of alarm rules from the specified system database. This system database can be used to store pre-configured alarm rules. The deployment location of the system database can be the server corresponding to the job monitoring system in this embodiment, a cloud server, or the management server of a cloud computing service platform. The rule reading interval can be a fixed value, meaning that alarm rules can be read from the specified system database periodically; or it can be a non-fixed value, meaning that the rule reading interval can be flexibly adjusted according to the actual operating environment or management needs.

[0087] S42, update the alarm rule set according to the read alarm rules to obtain a new alarm rule set.

[0088] In this embodiment, at predetermined rule reading intervals, the alarm rules contained in the specified system database are read once, and the alarm rule set generated based on the read alarm rules is used as the new alarm rule set. This enables dynamic updating of the currently used alarm rule set, thereby achieving dynamic updating of different alarm rules for multiple streaming jobs. This allows the monitoring and alarm service to be based on dynamically updated alarm rules, avoiding restarting alarm jobs and reducing the probability of missed alarms and untimely alarms caused by job restarts.

[0089] In some embodiments, the system database is a system database provided by the management server of the real-time computing platform; the system database is used to store: a set of alarm rules obtained by configuring the monitoring and alarm policies through the real-time computing platform; wherein, the monitoring and alarm policies include the monitoring and alarm policies for each streaming job of the first reporting link.

[0090] In this embodiment, at the management end of the real-time computing platform, the monitoring and alarm policies for each job are configured through the management page provided by the real-time computing platform. The configured alarm rules are stored in the system database provided by the management server of the real-time computing platform. Before the job monitoring system matches the monitoring indicator data to be matched with the pre-read alarm rule set, it can periodically read the alarm rules for each job from the system database and parse the read alarm rules into alarm rules that can be matched with the monitoring indicator data. This parsing may include parsing the alarm rule content and parsing the alarm rule data format, thereby providing the parsed alarm rules for alarm rule matching of the monitoring indicator data of the streaming job.

[0091] The above embodiments describe the specific implementation of monitoring metrics data for each data reporting link's streaming job, performing monitoring processing corresponding to the monitoring service type, and obtaining the corresponding monitoring processing results. To better understand the solution of this disclosure, the following describes, with examples, the specific improvements of this processing process to the processing logic of the native complex event processing component (Flink-CEP) in related technologies.

[0092] As an improvement, taking the alarm processing component based on Flink as an example, we will improve the method of receiving alarm rules in the native flink-cep.

[0093] Specifically, add a class to the function package in the native Flink-CEP component to receive the set of alarm rules.

[0094] For example, the function package in the native Flink-CEP component can be represented as org.apache.flink.cep.functions, which means a package of functions built on the native Flink-CEP based on the open source code provided by the open source organization (org.apache) for processing the receiving process of monitoring metric data.

[0095] For example, the added class for receiving the alarm rule set can be represented as InjectionPatternFunction, which means that the function package of this class is used to implement receiving (Injection) external alarm rule sets (Pattern).

[0096] For example, this class contains function code corresponding to two processing methods. One method can be represented as: the `Init()` method, used to initialize the connection of the external rule engine. The external rules can be a set of alarm rules read from a pre-defined system database, which is a system database provided by the management server of the real-time computing platform. The other method is the `Inject()` method, used to read the external rules and obtain the matching rules for all jobs. The alarm processing component of this embodiment calls the `Inject()` method and obtains the alarm rule set through the result returned by the method. For example, it can be called a Map set. Each record in the Map set can be represented as a "key-value" data format, such as a Map.<String,Pattern> Where String represents the job name and Pattern represents the alarm rule; the job name is the KEY and the alarm rule is the Value; the number of each job name is greater than or equal to 1, and the number of alarm rules for each job name is greater than or equal to 1.

[0097] As an improvement, a method function for receiving alarm rule sets has been added, which uses the received alarm rule sets to match the monitoring indicator data to be matched in the current reporting link.

[0098] Specifically, add a method to the event handling class in the native Flink-CEP component to receive a set of alarm rules.

[0099] For example, the event handling class in the native Flink-CEP component can be represented as org.apache.flink.cep.CEP, which means that it is a class (CEP) for performing complex event handling built on the native Flink-CEP based on the open source code provided by the open source organization (org.apache).

[0100] For example, the method for receiving alarm rule sets added in this class can be represented as InjectPattern, which means receiving and reading rules.

[0101] For example, the improved alarm processing component adds a method in the event processing class for receiving alarm rule sets. The method parameters include a real-time data stream and an anonymous inner class of the `InjectionPatternFunction` interface. The real-time data stream includes the monitoring metric data to be matched for the currently reported link, and the anonymous inner class represents the read alarm rule set. Through this method for receiving alarm rule sets, the monitoring metric data to be matched for the currently reported link is matched with the alarm rules in the read alarm rule set.

[0102] As an improvement, a determination has been added to determine whether the alarm rule set has been successfully read.

[0103] Specifically, in the rule set retrieval class of the native Flink-CEP component, a method is added to determine whether the read alarm rules meet the predetermined conditions. Specifically, the predetermined conditions can refer to: successfully reading the alarm rule set from the system database provided by the management server of the real-time computing platform.

[0104] For example, the rule collection retrieval class in the native Flink-CEP component can be represented as org.apache.flink.cep.PatternStream, which means that it is a class built for reading monitoring rules in the native Flink-CEP based on the open source code provided by the open source organization (org.apache).

[0105] For example, the constructor added to this class can be represented as the PatternStream() method. The parameters of this method include a real-time data stream, DataStream, and an anonymous inner class of the interface, InjectionPatternFunction. In this constructor, the DataStream's properties are defined as immutable (final), and the DataStream is defined to receive specific types of stream data. The constructor can also define the properties of the anonymous inner class, InjectionPatternFunction, as immutable. The real-time data stream includes: the monitoring indicator data to be matched for the currently reported link; the anonymous inner class represents: the set of alarm rules read. In this added constructor, the read monitoring indicator data to be matched and the read set of alarm rules are passed to the corresponding parent class constructor via the symbol (this). For example, this parent class constructor can be represented as PatternStreamBuilder.forStreamAndPattern(inputStream,function)). This means that when constructing the new `PatternStream()` method in the `PatternStreamBuilder` class, which builds the alarm rule set, the `InjectionPatternFunction` parameter can be checked to determine whether different event handling (CepOperator) logic is executed based on the result. If this parameter is not passed, the event handling logic in the native Flink-CEP component is executed, that is, the processing logic of matching a single data stream with a single alarm rule is executed; if this parameter is passed, it indicates that the alarm rule set has been successfully read, and the processing logic of matching the data stream with the alarm rule set is executed.

[0106] As an improvement, the rule matching method in the native Flink-CEP is improved.

[0107] As an example, in the rule matching class of the native Flink-CEP component, each member variable in the class is set as a Map.<KEY,Object> The format is as follows: KEY is the newly added key, representing the key-value pair of the data stream, and Object is the value, representing the original member variable type. In other words, by setting the correspondence between alarm rules and the key-value pairs of data streams, all alarm rules that need to be matched with monitoring metric data can be determined based on the key-value pair. This changes the existing processing logic of matching a single data stream with a single alarm rule, enabling the matching of data streams with multiple alarm matching rules.

[0108] In this example, based on the class corresponding to the rule matching method and the data structure of that class, if a data stream has multiple key values, it means that the data stream contains monitoring indicator data to be matched for multiple streaming jobs, thereby enabling the matching rules between multiple streaming jobs and the alarm rule set.

[0109] As an improvement, the method for periodically updating alarm matching rules in the native Flink-CEP is improved.

[0110] As an example, in the rule matching class of the native Flink-CEP component mentioned above, a timer is registered to periodically pull the latest rules from the outside through the Inject() method of the anonymous inner class InjectionPatternFunction, thereby realizing the dynamic update of the alarm rule set through the timed pull operation.

[0111] In the description of the above embodiments, the class name of the newly added class, the method name of the newly added method, and the parameter names of the parameters included in the newly added method can be customized, and this disclosure does not impose specific limitations.

[0112] The following specific embodiments describe the filtering scheme for monitoring indicators in this disclosure. By filtering the reported indicators, the data processing pressure of alarm monitoring is reduced and the system stability is improved.

[0113] In some embodiments, before matching the monitoring indicator data to be matched with the pre-read alarm rule set in step S12, the job monitoring method further includes the following steps.

[0114] S51 accesses the pre-defined storage space through the preset indicator reporting management interface and reads the set of indicator information to be reported from the pre-defined storage space.

[0115] In this step, the interface for the predetermined storage space can be provided by the data computing platform. Specifically, the data management platform can implement configuration management functions. For example, the data management platform provides a configuration management page, through which it receives the indicator information that needs to be reported configured by the user, and stores the indicator information that needs to be reported for configuration into the predetermined storage space, thus obtaining a set of indicator information for the indicators that need to be reported located in the predetermined storage space.

[0116] In some embodiments, the data computing platform may also provide an interface to access the predetermined storage space, through which information on all configured indicators that need to be reported can be obtained from the set of indicator information stored in the predetermined storage space.

[0117] S52, filter the monitoring indicator data to be matched based on the indicator information in the indicator information set, and use the filtered monitoring indicator data to be matched as the monitoring indicator data to be matched.

[0118] In this step, it is determined whether the indicator information of the monitoring indicator data to be matched is in the indicator information set. If the indicator information set contains the indicator information of the monitoring indicator data to be matched, it means that the indicator information is the indicator information that needs to be reported. Then, the monitoring indicator data to be matched corresponding to the indicator information is used as the filtered monitoring indicator data, which is used for subsequent monitoring processing corresponding to the monitoring alarm service on the filtered monitoring indicator data.

[0119] As an example, if the configured filtering metrics for the monitoring metrics to be matched only include job liveness, then consuming the Kafka topic (i.e., the topic corresponding to the monitoring metrics to be matched reported to the Kafka system) will only obtain the data for that monitoring metric.

[0120] According to the filtering process of monitoring metrics in this embodiment, the filtered monitoring metrics can reduce the processing pressure of the metric collection process and reduce the pressure on subsequent alarm processing, thereby improving the stability of alarm operations. If the received monitoring metric data from streaming jobs is reported to the Kafka system, it can not only reduce the overall pressure on the Kafka cluster, but also reduce the pressure on subsequent real-time alarm operations and improve the stability of alarm operations.

[0121] In some embodiments, the indicator reporting management interface is an interface provided by the management server of the real-time computing platform, and the predetermined storage space is a cache space used to cache the set of indicator information configured by the real-time computing platform at predetermined intervals. Step S51, the step of reading the set of indicator information to be reported from the predetermined storage space, may specifically include the following steps.

[0122] S61 reads a set of indicator information from the cache space.

[0123] In this embodiment, the cache space can be a memory space allocated by the real-time management platform to improve system performance. The main function of this cache space is to temporarily store the set of indicator information that needs to be reported, configured by the real-time management platform, in memory, and wait for the next access to the cache space.

[0124] S62, if the predetermined duration is less than the storage duration of the indicator information set in the cache space, then the read indicator information set is deduplicated, and the deduplicated indicator information set is used as the indicator information set of the indicators to be reported.

[0125] In this embodiment, the cache space can be a timed cache, i.e., it has a certain storage duration. This storage duration indicates the effective storage period of the indicator information set stored in the cache space. When the effective storage duration is exceeded, the content in the cache space will be cleared; furthermore, if the interval between two consecutive reads of the cache space is less than the storage duration, it indicates that the probability of information overlap between the read indicator information set and the previously read indicator information set is relatively high, therefore deduplication processing is required to reduce the redundancy of read information.

[0126] In some embodiments, the monitoring indicator data to be matched includes at least one of the following information items: indicator value (fields), indicator name (measurement), indicator unit of measurement (precision), indicator reporting time (time), and the operation information (tags) of the job to which the indicator belongs; the information items in the monitoring indicator data to be matched have a predetermined data format; the operation information of the job to which the indicator belongs includes at least one of the following information items: the hostname of the host on which the indicator is running, the job identifier (job_id) of the job to which the indicator belongs, and the job name (job_name).

[0127] Wherein, fields are the specific values ​​of the predefined metrics, which include, but are not limited to, at least one of the following metrics: job runtime, amount of data flowing in and out per unit time, job time consumption, CPU utilization, etc.; measurement is the metric name, precision is the description of the metric's unit of measurement. For example, when the metric is job runtime or job time consumption, the unit of measurement may include, but is not limited to, milliseconds, seconds, minutes, etc. The specific unit of measurement can be determined according to the actual situation; tags are the information records during the job's execution, such as recording the running information of the job's startup, execution, and destruction processes; host represents the name of the host where the job is running; job_id is the job identifier, used to identify the streaming job during the execution of the streaming computing framework (e.g., Flink); job_name is the job name of the streaming job during the execution of the streaming computing framework (e.g., Flink); time is the time when the metric is reported, that is, the event time of the streaming job in the streaming computing framework (e.g., Flink).

[0128] In some embodiments, the data type for the indicator reporting time can be a time type; the data type for the indicator name, the description information of the indicator unit of measurement, the information record during the operation of the job, the name of the host where the job is running, the job identifier, and the job name can be text, string, or other types. The specific data type can be customized according to the actual situation, and this disclosure does not impose any specific limitations.

[0129] In this embodiment, the data format can be used to filter information items in the monitoring indicator data to be matched. This filtering can retain the core data in the indicator data and reduce the redundancy of the monitoring indicator data. Furthermore, when the alarm processing component in this embodiment processes the monitoring indicator data to be matched that conforms to the data format, it can reduce the amount of data processing, save computing resources, and improve computing efficiency while realizing alarm services.

[0130] In this embodiment of the disclosure, the indicator data is reported to the Kafka system according to the agreed data format information. On the one hand, this can reduce the redundancy of monitoring indicator data and retain only the core data of the indicators. On the other hand, the monitoring service component consumes the monitoring indicator data reported to the Kafka system, thereby matching the monitoring indicator data reported to the Kafka system with the monitoring indicator set. While realizing the alarm business, it can also reduce the amount of data processing, save computing resources, and improve processing efficiency.

[0131] In some embodiments, the alarm rules in the alarm rule set are rules set based on streaming job events. Streaming job events include at least one of the following: job liveness event, job backpressure event, job automatic restart event, number of job checkpoint failures, message backlog event of job in the distributed publish-subscribe messaging system, and number of job task management recycling events.

[0132] In this embodiment, the rules set based on job survival events include: the job survival time within a predetermined statistical period reaches a survival time threshold; the rules set based on job backpressure events include: the number of job backpressure events occurring within a predetermined statistical period exceeds a job backpressure number threshold; the rules set based on job automatic restart events include: the number of job automatic restarts within a predetermined statistical period exceeds a restart number threshold; the rules set based on job checkpoint failure counts include: the number of job failures within a predetermined statistical period exceeds a failure number threshold; and the rules set based on taskManager Garbage Collection count events include: the number of collections within a predetermined statistical period exceeds a collection number threshold.

[0133] The following example uses rules set based on job liveness events to describe a specific instance of real-time monitoring and alarm implementation. The number of jobs, configured reporting metrics, monitoring metric data, and rule content in this example are illustrative and should not be construed as limiting the scope or feasibility of this solution.

[0134] As an example, the read alarm rule retrieved from the predefined system database might be based on the "job liveness" alarm rule. That is, the job monitoring alarm item configured on the real-time computing platform is job liveness, which depends on the job liveness duration (jobmanager_job_uptime) monitoring metric, and the job liveness duration threshold is set to 30 seconds. In this example, if monitoring metric data for the streaming job is received within the predefined statistical duration, it indicates that the streaming job is alive.

[0135] For example, the alarm processing component of this disclosure can be used to start a live Flink-CEP job to consume monitoring metric data reported to Kafka. For example, if job deletion (Cancel) is performed in the user interface of the Flink framework, an alarm message indicating that the job is alive will be received within 30 seconds, thus realizing the monitoring alarm for the monitoring metric data.

[0136] For example, using the alarm processing component of this disclosure embodiment, two streaming jobs, such as job A and job B, are started in the resource management system. The alarm rules for job A are based on an alarm rule (survival policy) set according to "job survival," with a survival time threshold set, for example, to 30 seconds. If no alarm indicator data is received from job A, it can be determined that the alarm rule matching for job A has failed, and no alarm data is generated. Similarly, the alarm rules for job B are also based on an alarm rule (survival policy) set according to "job survival," with a survival time threshold set, for example, to 40 seconds. If no alarm indicator data is received from job B, it can be determined that the alarm rule matching for job B has failed, and no alarm data is generated.

[0137] In this example, if job A and job B are deleted separately, job A will generate an alarm message 30 seconds after deletion; job B will generate an alarm message 40 seconds after deletion. Then, jobs A and B are restarted, and the survival policy (survival time threshold set to 40 seconds) for both jobs A and B is set. Five minutes later, jobs A and B are deleted again. Job A receives an alarm 40 seconds after deletion, and job B receives an alarm 30 seconds after deletion. During alarm processing, the alarm rule matching for both job A and job B is successful.

[0138] As can be seen from the above examples, the monitoring rule matching of this disclosure embodiment supports multiple rules and dynamic updates, providing real-time dynamic monitoring and alarm services for received streaming jobs.

[0139] The following embodiments describe specific implementation methods for providing monitoring metric visualization services through monitoring metric data. In the following embodiments, monitoring processing corresponding to the monitoring metric visualization service is performed on the monitoring metric data.

[0140] In some embodiments, at least one data reporting link includes a second reporting link corresponding to the monitoring metric visualization service; step S120 may specifically include the following steps.

[0141] S71, take the monitoring indicator data of the streaming operation of the second reporting link as the monitoring indicator data to be displayed;

[0142] S72, Obtain monitoring indicator data that meets the predetermined filtering conditions from the monitoring indicator data to be displayed;

[0143] S73 sends the monitoring indicator data that meets the screening criteria to the predetermined data visualization platform so that the monitoring indicator data that meets the screening criteria can be displayed through the data visualization platform.

[0144] Through the above steps S71-S73, the filtering conditions are used to obtain the monitoring indicator data that needs to be displayed from the monitoring indicator data to be displayed, and to visualize the monitoring indicators.

[0145] In some embodiments, a data visualization platform can be used to visualize monitoring data. Exemplary examples include, but are not limited to, the cross-platform open-source metrics analysis and visualization tool Grafana, the machine data engine Splunk, and the data visualization and mining tool Kibana. Grafana allows users to query and visualize collected data, and generates timely notification messages for the displayed content; Splunk collects, indexes, and utilizes fast mobile computing data generated by all applications, servers, and devices, and visualizes it.

[0146] In some embodiments, the data visualization platform includes a job monitoring page, which contains multiple page tags, each corresponding to a filter condition. Step S73, which involves displaying monitoring indicator data that meets the filter conditions through the data visualization platform, may specifically include: on the job monitoring page, displaying monitoring indicator data that meets the corresponding filter conditions according to the filter conditions corresponding to each page tag.

[0147] To facilitate understanding, through Figure 4 This diagram illustrates a job monitoring page according to an embodiment of the present disclosure. In some embodiments, the job monitoring page may include multiple tabs. Figure 4The example shows the following tags: Job Name, Real-time Data Warehouse (Doris), Operator Name, Task Manager ID, and Job Manager; and the example shows the following tags in the list: Overview Information, Job Management Information (Virtual Machine), Job Management (Checkpoint), Task Management (Virtual Machine), and Task Management (Task Computation Task). Figure 4 Each label in the system is used to display data for the corresponding metric; when each label is expanded, it can display further subdivided metrics under that label; different levels of label classification are implemented, so that monitoring data classified according to each label can be viewed, and the hierarchical classification of metrics can be displayed in the visualization interface.

[0148] refer to Figure 4 The label information may include the job name; that is, in some embodiments, the filtering conditions are set based on the attribute information of the monitoring indicator data, and the attribute information includes the job name. Embodiments of this disclosure support filtering and visualizing the monitoring indicator data to be displayed based on the job name.

[0149] In this embodiment, prior to step S72, the job monitoring method further includes the following steps.

[0150] S81, retrieve the job name corresponding to the monitoring indicator data to be displayed.

[0151] S82, if the obtained job name is empty, then read the operating environment information of the monitoring indicator data to be displayed from the predetermined parameter transmission path.

[0152] S83 reads the job name from the runtime environment information.

[0153] In this embodiment, the method for obtaining the job name is the same as the method for obtaining the job name described in conjunction with S21 in the previous embodiment, and will not be repeated here.

[0154] For example, in Yarn, during the startup process of a Flink job, parameters are set through a job configuration class. The job configuration class can be represented as org.apache.flink.configuration.Configuration, which means: based on the open source code provided by the open source organization (org.apache), the corresponding class configures the Flink job.

[0155] As an example of a parameter setting function: "env.java.opts" = "-DjobName" + jobName, where env.java.opts represents the name of the function implementing the parameter, jobName is the job name field, and -DjobName is the content of the job name field.

[0156] As an example, when processing the reporting of monitoring metric data to be displayed, before receiving the metric data from the resource management system and before each report, it is determined whether the metric data contains a job name field. If the job name field is not present, the job name is obtained from the virtual machine environment corresponding to the streaming job in Yarn. The function for obtaining the job name can be represented as System.getProperties(“jobName”). Through the above processing, each monitoring metric data to be displayed can be guaranteed to include a job name.

[0157] According to the job monitoring method of this disclosure, each streaming computing task (i.e., streaming job) running in the resource management system can be assigned a job name. During the reporting of monitoring indicators for streaming jobs, the resource management system does not always upload the job name as a reporting indicator field based on the preset monitoring indicators supported by the streaming computing framework itself. However, in this disclosure, the resource management system can support uploading the streaming job name as an upload field when the monitoring indicator data of the streaming task is transmitted to the job monitoring system. If it is not uploaded, it can also be obtained from the running environment of the streaming job through a pre-provided interface. Therefore, during monitoring visualization, it can support filtering the visualized data of each monitoring indicator by job name.

[0158] In some embodiments, after step S72 and before step S73, the job monitoring method further includes the following steps.

[0159] S91, using the proxy service of the time series database, the monitoring indicator data to be displayed is allocated to at least one time series database according to the predetermined allocation rules; and S92, using the proxy service of the time series database, the monitoring indicator data that meets the filtering conditions is obtained from at least one time series database according to the predetermined allocation rules.

[0160] In this embodiment of the disclosure, when storing the monitoring data to be displayed into at least one time-series database, the high availability of the time-series database can be achieved through a proxy service of the time-series database. Storing the monitoring indicator data to be displayed in the time-series database allows for trend evaluation of the data, which is beneficial for timely problem detection and subsequent corrective action. The proxy service of the time-series database is a cluster proxy service based on the time-series database, which can realize a highly available cluster deployment of the time-series database, and thus has the capabilities of dynamic scaling up / down, fault recovery, and data synchronization.

[0161] In some embodiments, the predetermined allocation rule is a rule set based on at least one information item in the monitoring indicator data to be displayed. In this embodiment, the monitoring indicator data to be displayed is classified according to the execution information item, and the classified monitoring indicator data is stored in a time-series database through a proxy server that implements the time-series database, thereby realizing distributed storage of the monitoring indicator data to be displayed.

[0162] According to the job allocation method of the above embodiments of this disclosure, at least one data interface of a data reporting link can be pre-configured for the received monitoring index data of streaming jobs. Transmitting the data to any data reporting link means that a corresponding monitoring service type has been selected for the monitoring index data, so that a monitoring service of the corresponding monitoring server type can be implemented according to the received monitoring index data of streaming jobs.

[0163] Figure 5 A schematic diagram of the monitoring architecture system according to an embodiment of this disclosure is shown. Figure 5 The monitoring architecture system may include: a real-time data computing platform 510, a resource management system 520, and a job monitoring system 530.

[0164] The real-time data computing platform 510 is used to submit streaming jobs to the resource management system; the resource management system 520 is used to run streaming jobs and send the monitoring index data generated by the running streaming jobs to the job monitoring system; the job monitoring system 530 is used to receive the streaming job monitoring index data and execute the job monitoring method of the above embodiment.

[0165] Figure 5 The real-time data computing platform 510 and Figure 2 The real-time data computing platform 210 in China Figure 5 Resource Management System 520 and Figure 2 Resource management system 220, and Figure 5 The 530 operation monitoring system in the middle and Figure 2The operation monitoring system 230 in the above embodiments has the same structure and can achieve the same functions. It should be clarified that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods and modules are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0166] According to the monitoring architecture system of this disclosure, different monitoring service types are provided for the monitoring indicator data of streaming operations, thereby transmitting the monitoring indicator data to the data interfaces of different reporting links and executing the corresponding monitoring services, thus achieving diversification of monitoring service types. By performing different monitoring processing on the monitoring indicator data of different reporting links, it is beneficial to subsequently execute the monitoring processing corresponding to each monitoring service type synchronously, thereby improving the monitoring processing efficiency. It is also beneficial to quickly locate the problematic data reporting link when system errors of different monitoring service types occur, thereby improving the efficiency of problem investigation.

[0167] In some embodiments, the monitoring architecture system further includes a data visualization platform 250; wherein, the data visualization platform 250 is used to provide a preset resource location address for the monitoring visualization page, the resource location address being used to establish an external link from the real-time computing platform 210 to the monitoring visualization page; the real-time computing platform 210 is also used to provide a job alarm management page, and when the job alarm management page detects a jump operation pointing to the monitoring visualization page, it sends a jump request to the monitoring visualization page through the external link; the data visualization platform 250 is also used to display streaming job monitoring indicator data through the monitoring visualization page based on the jump request.

[0168] In this embodiment, the real-time computing platform can jump from its own job alarm management page to the data visualization platform page, enabling flexible switching between pages.

[0169] Figure 6 This diagram illustrates the page before navigation in a data visualization platform that embodies an exemplary embodiment of this disclosure. Figure 6 The image displayed is a monitoring visualization page of the data visualization platform. By clicking on the "Work Curve" on the monitoring visualization page, you can jump to the page corresponding to that work curve on the data visualization platform.

[0170] It should be understood that the content on a monitoring visualization page of this data visualization platform is merely illustrative and does not affect the scope of protection of the embodiments disclosed herein.

[0171] The following is through Figure 7 and Figure 8This document describes a job monitoring system and job monitoring method according to exemplary embodiments of the present disclosure.

[0172] Figure 7 A schematic diagram of the structure of a job monitoring system according to an exemplary embodiment of the present disclosure is shown; Figure 8 A flowchart illustrating a job monitoring method according to an exemplary embodiment of this disclosure is provided. Figure 7 and Figure 2 and Figure 5 The same label indicates the same structure.

[0173] exist Figure 7 In the process, the job monitoring system 230 also includes a message transmission system reporter 71, a time-series data reporter 72, and a rule matching component 73; Figure 7 The diagram also illustrates an alarm service platform 240, a visualization platform 250, and a time-series database proxy server 260 that provide proxy services for the time-series data cluster 270. The time-series data cluster 270 contains multiple time-series databases, two of which, for example... Figure 7 The time series database 1 and time series database 2 are shown in the figure.

[0174] exist Figure 7 In the operation monitoring system, multiple monitoring indicator uploaders can be configured, such as message transmission system uploader 71 and time series data uploader 72. Message transmission system uploader 71 can be used for real-time alarm of operation monitoring indicators, supports dynamic filtering of reported monitoring data indicators, and displays the indicator information of the filtered monitoring data indicators through the visualization platform 250.

[0175] exist Figure 7 In this process, a rule matching component 73, modified from the native complex event processing component, can be used to match multiple alarm rules on the monitoring indicator data reported by the message transmission system reporter 71 to perform real-time alarms for streaming jobs. In this matching process, the monitoring indicator data of each streaming job corresponds to the matching of multiple rules. The time series data reporter 72 can be used for the visualization of job monitoring data indicators and can use the proxy service provided by the time series database proxy server 260 to achieve high availability of the time series data cluster 270.

[0176] exist Figure 7 In the process, the rule matching results of the rule matching component 73 are uploaded to the real-time computing platform 210. If there are target matching rules for the monitoring indicator data, alarm information can be output through the alarm service platform 240.

[0177] exist Figure 7In the process, the time-series database cluster 270 can be used to receive and store the monitoring indicator data reported by the job monitoring system 230, providing data storage guarantee for subsequent querying and visualization processing of the monitoring indicator data; the alarm service platform 240 can be used to receive alarm information obtained by the job monitoring system 230 from monitoring indicator data, and realize the output of alarm information; the visualization platform 250 is used to realize the visualization of monitoring indicator data based on the visualization display requirements of the job monitoring system 230 for monitoring indicator data.

[0178] like Figure 8 As shown, the job monitoring method may include the following steps.

[0179] S801, a real-time computing platform, generates streaming jobs and configures alarm rules.

[0180] S802, Submit streaming jobs to the resource management system.

[0181] S803, the resource management system runs streaming jobs.

[0182] If the resource management system fails to run the streaming job in this step, no indicators will be reported; if the operation is successful, proceed to step S811.

[0183] S811 receives monitoring metrics data from streaming operations.

[0184] S812, the monitoring index data of the streaming operation is reported to the message transmission system reporter, and steps S812-S819 and steps S831-S832 are executed.

[0185] Step S821: Report the monitoring index data of the streaming operation to the time series data reporter, and execute steps S821-S825.

[0186] S813 dynamically filters monitoring indicator data.

[0187] S814 reports the dynamically filtered monitoring indicator data to the message transmission system.

[0188] S815 retrieves alarm rules and dynamically updates the alarm rule set.

[0189] S830, determine whether the time interval since the last time an alarm rule was read from the specified system database has reached the preset rule reading interval.

[0190] S831, if the rule reading interval is reached, the alarm rule is updated, and S816 is executed based on the updated alarm rule.

[0191] S832, if the rule reading interval time has not been reached, the alarm rule will not be updated, and step S816 will be executed according to the current alarm rule.

[0192] S816 matches the reported monitoring metric data with the alarm rule set.

[0193] S817 If the target alarm rule of the reported monitoring indicator data is matched, the alarm time is confirmed to be successfully matched.

[0194] S818 sends alarm messages to the real-time computing platform.

[0195] S822 receives monitoring metrics data from streaming jobs and sends it to the time-series database's proxy server.

[0196] S823, the time series database's proxy server distributes monitoring metric data from streaming jobs to the time series database.

[0197] The S824 is a visualization platform for monitoring indicator data and can provide external links for real-time calculation of work curves.

[0198] When S825 receives a request to jump to the real-time calculation job curve, it displays the page for the real-time calculation job curve.

[0199] According to the business monitoring method of this disclosure, the real-time computing platform submits streaming jobs, which is beneficial for the streaming jobs to run normally in the resource management system. After the jobs run normally, the resource management system reports the monitoring index data of the jobs.

[0200] In the disclosed embodiments, the operation monitoring index data can be uploaded to the proxy server of the time series database. After proxy calculation, the monitoring index data is distributed to the time series database instance. The time series database can be connected to the visualization platform through the proxy server of the time series database, so that visualization pages or visualization dashboards can be created on the visualization platform to realize the visualization display of monitoring index data.

[0201] In the disclosed embodiments, the job monitoring index data can also be uploaded to the messaging system to match the monitoring index data of the streaming job with multiple alarm rules in the alarm rule set. If the monitoring index data uploaded to the messaging system matches the target alarm rule, an alarm message is generated according to the matched target alarm rule as the monitoring processing result corresponding to the alarm message.

[0202] According to the job monitoring method of this disclosure, different monitoring service types are provided for the monitoring indicator data of streaming jobs, thereby transmitting the monitoring indicator data to the data interfaces of different reporting links and executing the corresponding monitoring services, thus achieving diversification of monitoring service types. By performing different monitoring processing on the monitoring indicator data of different reporting links, it is beneficial to subsequently execute the monitoring processing corresponding to each monitoring service type synchronously, thereby improving the monitoring processing efficiency. It is also beneficial to quickly locate the problematic data reporting link when system errors of different monitoring service types occur, thereby improving the efficiency of problem investigation.

[0203] The service monitoring method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0204] In addition, this disclosure also provides a job monitoring method, apparatus, electronic device, and computer-readable storage medium, all of which can be used to implement any of the job monitoring methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section on methods and will not be repeated here.

[0205] Figure 9 This is a block diagram of a work monitoring device provided in an embodiment of the present disclosure.

[0206] Reference Figure 9 This disclosure provides a job monitoring device, which includes: a transmission module 910, used to transmit received monitoring index data of streaming jobs to the data interface of at least one pre-configured data reporting link; wherein different data reporting links correspond to different monitoring service types; and a processing module 920, used to perform monitoring processing corresponding to the monitoring service type on the monitoring index data of streaming jobs of each data reporting link, and obtain the corresponding monitoring processing result.

[0207] In some embodiments, different data reporting links include a first reporting link and a second reporting link; different monitoring service types include: a monitoring alarm service corresponding to the first reporting link and a monitoring indicator visualization service corresponding to the second reporting link; the transmission module 910 is specifically used to transmit the received monitoring indicator data to the data interface of the first reporting link and the data interface of the second reporting link, so as to provide the corresponding monitoring alarm service and the corresponding monitoring indicator visualization service.

[0208] In some embodiments, the data interface of the first reporting link is a data interface corresponding to the distributed publish-subscribe messaging system; the data interface of the second reporting link is a data interface corresponding to the time-series database.

[0209] In some embodiments, at least one data reporting link includes a first reporting link corresponding to the monitoring and alarm service; the processing module 920 is specifically configured to: use the monitoring indicator data of the streaming job of the first reporting link as the monitoring indicator data to be matched; match the monitoring indicator data to be matched with a pre-read alarm rule set; wherein, the alarm rule set includes at least one alarm rule, and each alarm rule corresponds to the job name of each streaming job; if the monitoring indicator data to be matched matches a target alarm rule from the alarm rule set, then generate alarm information according to the target alarm rule, and use the alarm information as the corresponding monitoring processing result.

[0210] In some embodiments, the monitoring indicator data to be matched corresponds to at least two streaming jobs of the first reporting link; the processing module 920 is used to match the monitoring indicator data to be matched with a pre-read alarm rule set, specifically for: obtaining the job name of the monitoring indicator data to be matched; grouping the monitoring indicator data to be matched based on the job name; obtaining the corresponding alarm rule from the alarm rule set according to the job name of the monitoring indicator data to be matched in each group, to obtain a subset of alarm rules corresponding to each group; matching the monitoring indicator data to be matched of each streaming job of the first reporting link with the subset of alarm rules corresponding to its group, to obtain a target alarm rule that matches the monitoring indicator data to be matched of each streaming job of the first reporting link.

[0211] In some embodiments, each alarm rule in the same subset of alarm rules corresponds to the same preset key value, and the same preset key value is used to identify the same monitoring indicator data to be matched; the processing module 920, when obtaining the corresponding alarm rule from the alarm rule set according to the job name of the monitoring indicator data to be matched for each group, specifically performs the following: obtaining the key value of the monitoring indicator data to be matched for each streaming job of the first reporting link; and using the subset of alarm rules corresponding to the key value of the monitoring indicator data to be matched as the subset of alarm rules corresponding to each group.

[0212] In some embodiments, before matching the monitoring indicator data to be matched with the pre-read alarm rule set, the processing module 920 is also used to read alarm rules from a specified system database at predetermined rule reading intervals; and to update the alarm rule set according to the read alarm rules to obtain a new alarm rule set.

[0213] In some embodiments, the system database is a system database provided by the management server of the real-time computing platform; the system database is used to store: a set of alarm rules obtained by configuring the monitoring and alarm policies through the real-time computing platform; wherein, the monitoring and alarm policies include the monitoring and alarm policies for each streaming job of the first reporting link.

[0214] In some embodiments, before matching the monitoring indicator data to be matched with the pre-read alarm rule set, the processing module 920 is further configured to: access a predetermined storage space through a preset indicator reporting management interface, read the indicator information set of the indicators to be reported from the predetermined storage space; filter the monitoring indicator data to be matched according to the indicator information in the indicator information set, and use the filtered monitoring indicator data to be matched as the monitoring indicator data to be matched.

[0215] In some embodiments, the indicator reporting management interface is an interface provided by the management server of the real-time computing platform. The predetermined storage space is a cache space, which is used to cache the indicator information set configured by the real-time computing platform at predetermined intervals. The processing module 920 is used to read the indicator information set of the indicators to be reported from the predetermined storage space, specifically to read the indicator information set from the cache space. If the predetermined interval is less than the storage time of the indicator information set in the cache space, the read indicator information set is deduplicated, and the deduplicated indicator information set is used as the indicator information set of the indicators to be reported.

[0216] In some embodiments, the monitoring indicator data to be matched includes at least one of the following information items: indicator value, indicator name, indicator unit of measurement, indicator reporting time, and operation information of the job to which the indicator belongs; the information items in the monitoring indicator data to be matched have a predetermined data format; the operation information of the job to which the indicator belongs includes at least one of the following information items: hostname of the host on which the indicator is running, job identifier of the job to which the indicator belongs, and job name.

[0217] In some embodiments, the alarm rules in the alarm rule set are rules set based on streaming job events. Streaming job events include at least one of the following: job liveness event, job backpressure event, job automatic restart event, number of job checkpoint failures, message backlog event of job in the distributed publish-subscribe messaging system, and number of job task management recycling events.

[0218] In some embodiments, at least one data reporting link includes a second reporting link corresponding to the monitoring indicator visualization service; the processing module 920 is specifically used to take the monitoring indicator data of the streaming operation of the second reporting link as the monitoring indicator data to be displayed; obtain the monitoring indicator data that meets the predetermined filtering conditions from the monitoring indicator data to be displayed; and send the monitoring indicator data that meets the filtering conditions to the predetermined data visualization platform so as to display the monitoring indicator data that meets the filtering conditions through the data visualization platform.

[0219] In some embodiments, the data visualization platform includes a job monitoring page, which contains multiple page labels, each corresponding to a filter condition. The processing module 920 is specifically used to display monitoring indicator data that meet the filter conditions through the data visualization platform. Specifically, on the job monitoring page, it displays monitoring indicator data that meet the corresponding filter conditions according to the filter conditions corresponding to each page label.

[0220] In some embodiments, the filtering conditions are set based on the attribute information of the monitoring indicator data, and the attribute information includes the job name; before the processing module 920 obtains the monitoring indicator data that meets the predetermined filtering conditions from the monitoring indicator data to be displayed, it is also used to obtain the job name corresponding to the monitoring indicator data to be displayed; if the obtained job name is empty, the operating environment information of the monitoring indicator data to be displayed is read from the predetermined parameter transmission path; and the job name is read from the operating environment information.

[0221] In some embodiments, the processing module 920 is further configured to, after acquiring monitoring indicator data that meets predetermined filtering conditions, and before sending the monitoring indicator data that meets the filtering conditions to a predetermined data visualization platform, use a time series database proxy service to allocate the monitoring indicator data to be displayed to at least one time series database according to predetermined allocation rules; and use the time series database proxy service to acquire monitoring indicator data that meets the filtering conditions from at least one time series database according to predetermined allocation rules.

[0222] In some embodiments, the predefined allocation rule is a rule set based on at least one information item in the monitoring metric data to be displayed.

[0223] According to the operation monitoring device of this disclosure, different monitoring service types are provided for the monitoring indicator data of streaming operations, thereby transmitting the monitoring indicator data to the data interfaces of different reporting links and executing the corresponding monitoring services, thus realizing the diversification of monitoring service types. By performing different monitoring processing on the monitoring indicator data of different reporting links, it is beneficial to subsequently execute the monitoring processing corresponding to each monitoring service type synchronously, thereby improving the monitoring processing efficiency. It is also beneficial to quickly locate the problematic data reporting link when system errors of different monitoring service types occur, thereby improving the efficiency of problem investigation.

[0224] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0225] Figure 10 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0226] Reference Figure 10 This disclosure provides an electronic device, which includes: at least one processor 1001; at least one memory 1002; and one or more I / O interfaces 1003 connected between the processor 1001 and the memory 1002; wherein the memory 1002 stores one or more computer programs that can be executed by the at least one processor 1001, and the one or more computer programs are executed by the at least one processor 1001 to enable the at least one processor 1001 to perform the above-described job monitoring method.

[0227] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the job monitoring method described above. The computer-readable storage medium may be volatile or non-volatile.

[0228] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described job monitoring method.

[0229] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0230] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0231] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0232] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0233] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0234] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0235] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0236] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0237] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0238] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A method for monitoring operations, characterized in that, include: The received monitoring metrics data of the streaming job are transmitted to the data interface of at least one pre-configured data reporting link; wherein, different data reporting links correspond to different monitoring service types, including a first reporting link and a second reporting link, and different monitoring service types include the monitoring alarm service corresponding to the first reporting link and the monitoring metric visualization service corresponding to the second reporting link. For the monitoring metrics data of each data reporting link's streaming job, perform monitoring processing corresponding to the monitoring service type to obtain the corresponding monitoring processing results.

2. The monitoring method according to claim 1, characterized in that, The different data reporting links include a first reporting link and a second reporting link; The different monitoring service types include: the monitoring alarm service corresponding to the first reporting link and the monitoring indicator visualization service corresponding to the second reporting link; The step of transmitting the received monitoring metric data of the streaming job to the data interface of at least one pre-configured data reporting link includes: The received monitoring indicator data is transmitted to the data interface of the first reporting link and the data interface of the second reporting link to provide the corresponding monitoring alarm service and the monitoring indicator visualization service.

3. The monitoring method according to claim 2, characterized in that, The data interface of the first reporting link is the data interface corresponding to the distributed publish-subscribe messaging system; the data interface of the second reporting link is the data interface corresponding to the time-series database.

4. The monitoring method according to claim 1, characterized in that, The at least one data reporting link includes a first reporting link corresponding to the monitoring and alarm service; the monitoring indicator data of the streaming job for each data reporting link is subjected to monitoring processing corresponding to the monitoring service type to obtain the corresponding monitoring processing result, including: The monitoring index data of the streaming job in the first reporting link is used as the monitoring index data to be matched. The monitoring indicator data to be matched is matched with a pre-read alarm rule set; wherein, the alarm rule set includes at least one alarm rule, and each alarm rule corresponds to the job name of each streaming job; If the monitoring indicator data to be matched matches the target alarm rule in the alarm rule set, then alarm information is generated according to the target alarm rule, and the alarm information is used as the corresponding monitoring processing result.

5. The monitoring method according to claim 4, characterized in that, The monitoring metric data to be matched corresponds to at least two streaming jobs in the first reporting link; the matching of the monitoring metric data to be matched with the pre-read alarm rule set includes: The job name for obtaining the monitoring indicator data to be matched; Group the monitoring indicator data to be matched based on the job name; Based on the job name of the monitoring indicator data to be matched for each group, the corresponding alarm rule is obtained from the alarm rule set to obtain the alarm rule subset corresponding to each group; The monitoring metric data to be matched for each streaming job in the first reporting link is matched with the alarm rule subset corresponding to the group to which it belongs, so as to obtain the target alarm rule that matches the monitoring metric data to be matched for each streaming job in the first reporting link.

6. The monitoring method according to claim 5, characterized in that, Each alarm rule in the same subset of alarm rules corresponds to the same preset key value, which is used to identify the same monitoring indicator data to be matched; The step of obtaining the corresponding alarm rule from the alarm rule set based on the job name of the monitoring indicator data to be matched for each group, and obtaining the alarm rule subset corresponding to each group, includes: Obtain the key value of the monitoring indicator data to be matched for each streaming job in the first reporting link; The subset of alarm rules corresponding to the key values ​​of the monitoring indicator data to be matched is taken as the subset of alarm rules corresponding to each group.

7. The monitoring method according to claim 4, characterized in that, Before matching the monitoring metric data to be matched with the pre-read alarm rule set, the method further includes: Access the predetermined storage space through the preset indicator reporting management interface, and read the set of indicator information of the indicators to be reported from the predetermined storage space; The monitoring indicator data to be matched is filtered according to the indicator information in the indicator information set, and the filtered monitoring indicator data to be matched is used as the monitoring indicator data to be matched.

8. The monitoring method according to claim 7, characterized in that, The indicator reporting management interface is an interface provided by the management server of the real-time computing platform. The predetermined storage space is a cache space, which is used to cache the set of indicator information configured by the real-time computing platform at predetermined intervals. The step of reading the set of indicator information to be reported from the predetermined storage space includes: Read the set of indicator information from the cache space; If the predetermined duration is less than the storage duration of the indicator information set in the cache space, then the read indicator information set is deduplicated, and the deduplicated indicator information set is used as the indicator information set of the indicator to be reported.

9. The monitoring method according to claim 1, characterized in that, The at least one data reporting link includes a second reporting link corresponding to the monitoring metric visualization service; the monitoring metric data of the streaming job for each data reporting link is processed according to the monitoring service type to obtain the corresponding monitoring processing result, including: The monitoring metrics data of the streaming job in the second reporting link will be used as the monitoring metrics data to be displayed. From the monitoring indicator data to be displayed, obtain monitoring indicator data that meets the predetermined filtering conditions; The monitoring indicator data that meets the filtering criteria is sent to a predetermined data visualization platform so that the monitoring indicator data that meets the filtering criteria can be displayed through the data visualization platform.

10. The monitoring method according to claim 9, characterized in that, After obtaining monitoring indicator data that meets predetermined filtering criteria, and before sending the monitoring indicator data that meets the filtering criteria to a predetermined data visualization platform, the method further includes: Using the proxy service of the time series database, the monitoring indicator data to be displayed is allocated to at least one time series database according to the predetermined allocation rules; In addition, using the proxy service of the time series database, monitoring indicator data that meet the filtering conditions are obtained from the at least one time series database according to the predetermined allocation rules.

11. A work monitoring device, characterized in that, include: The transmission module is used to transmit the received monitoring indicator data of the streaming job to the data interface of at least one pre-configured data reporting link; wherein, different data reporting links correspond to different monitoring service types, the different data reporting links include a first reporting link and a second reporting link, and the different monitoring service types include the monitoring alarm service corresponding to the first reporting link and the monitoring indicator visualization service corresponding to the second reporting link. The processing module is used to perform monitoring processing corresponding to the monitoring service type on the monitoring indicator data of each data reporting link's streaming job, and obtain the corresponding monitoring processing results.

12. A monitoring architecture system, characterized in that, include: Real-time data computing platform, resource management system, and job monitoring system; The real-time data computing platform is used to submit streaming jobs to the resource management system; The resource management system is used to run the streaming job and send the monitoring index data generated by the streaming job to the job monitoring system. The job monitoring system is used to receive the streaming job monitoring index data and execute the job monitoring method according to claims 1-10.

13. The system according to claim 12, characterized in that, The monitoring architecture system also includes a data visualization platform; wherein... The data visualization platform is used to provide a preset resource location address for the monitoring visualization page, and the resource location address is used to establish an external link from the real-time computing platform to the monitoring visualization page. The real-time computing platform is also used to provide a job alarm management page, and when the job alarm management page detects a jump operation pointing to the monitoring visualization page, it sends a jump request to the monitoring visualization page through the external link; The data visualization platform is also used to display the streaming job monitoring indicator data through the monitoring visualization page based on the jump request.

14. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the monitoring method as described in any one of claims 1-10.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the monitoring method as described in any one of claims 1-10.

Images