Real-time streaming application monitoring method and device, medium and product

Abstract

The invention discloses a real-time streaming application monitoring method and device, a medium and a product. The method comprises the following steps: positioning a target operator according to configuration information; in the running process, when a trigger instruction is detected every time, the generation time of the trigger instruction is used as starting time, and the accumulated data processing quantity and the accumulated processing time are monitored in real time; when the accumulated data processing quantity is greater than or equal to a quantity threshold value, or the accumulated processing time is greater than or equal to a time threshold value, ending the monitoring process, constructing a monitoring period according to the ending time and the starting time, calculating the number of transactions per second and the delay time in the monitoring period, storing the transactions per second and the delay time in a distributed cache, and generating a new trigger instruction; according to the transaction number per second and the time delay time in the cache, a transaction number per second performance curve and a time delay performance curve are obtained and displayed, visual display of the performance is achieved, through distributed cache, the production operation risk caused by random address change after the real-time streaming application is restarted is reduced, and the monitoring stability and reliability are improved.

Description

Technical Field

[0001] The present invention relates to the fields of Internet, cloud computing and big data service technology, and in particular to a monitoring method, device, medium and product for real-time streaming applications. Background Art

[0002] With the development of internet technology, data is growing exponentially. To extract valuable information from this massive amount of data, a large number of real-time computing frameworks capable of rapidly processing this vast amount of data have emerged. As a key component of data processing, the performance of real-time streaming applications directly determines the efficiency and effectiveness of data processing.

[0003] Existing technologies use the built-in monitoring functionality of real-time streaming applications to view operator data metrics. This approach fails to monitor the overall processing latency of job data, reducing the reliability of real-time streaming application status monitoring. Furthermore, current performance monitoring of real-time streaming applications focuses primarily on file operations, such as file uploads, downloads, and modifications, rather than the overall application status and performance. Furthermore, after a job in a real-time streaming application is restarted, its monitoring address often changes, resulting in monitoring failures and risks to application operation, reducing the stability and reliability of real-time streaming application status monitoring. Summary of the Invention

[0004] The present invention provides a monitoring method, device, medium and product for real-time streaming applications, so as to solve the problems of low stability and poor reliability of real-time streaming application status monitoring.

[0005] According to one aspect of an embodiment of the present invention, a method for monitoring a real-time streaming application is provided, comprising:

[0006] Locating, in the big data cluster, a target operator in at least one target real-time stream job to be monitored according to the real-time stream application configuration information to be monitored;

[0007] During the execution of the target real-time streaming job, whenever a trigger instruction is detected, the cumulative data processing amount and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated;

[0008] When the cumulative data processing volume of the target operator is greater than or equal to the preset volume threshold, or the cumulative processing time of the target operator is greater than or equal to the preset time threshold, the monitoring process ends and a monitoring cycle is established based on the process end time and start time.

[0009] Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second and latency during the monitoring period are calculated and stored in the distributed cache, and a new trigger instruction is generated;

[0010] Based on the number of transactions per second and latency of the target operator stored in the distributed cache in different monitoring cycles, the target operator's transaction number per second performance curve and latency performance curve are obtained for display.

[0011] According to another aspect of an embodiment of the present invention, a monitoring device for a real-time streaming application is provided, comprising:

[0012] A positioning operator module is used to locate a target operator in at least one target real-time stream job to be monitored in the big data cluster according to the configuration information of the real-time stream application to be monitored;

[0013] The real-time monitoring module is used to monitor the cumulative data processing amount and cumulative processing time of the target operator in real time whenever a trigger instruction is detected during the operation of the target real-time streaming job, starting from the time when the trigger instruction is generated;

[0014] The monitoring cycle module is used to end the monitoring process when the cumulative data processing amount of the target operator is greater than or equal to the preset number threshold, or the cumulative processing time of the target operator is greater than or equal to the preset time threshold, and to establish a monitoring cycle based on the process end time and start time;

[0015] The calculation and storage module is used to calculate the number of transactions per second and latency time during the monitoring period based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, store them in the distributed cache, and trigger the generation of new trigger instructions;

[0016] The curve display module is used to obtain the target operator's transaction number per second performance curve and latency performance curve for display based on the target operator stored in the distributed cache in different monitoring cycles.

[0017] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the real-time streaming application monitoring method described in any embodiment of the present invention.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the monitoring method of real-time streaming applications according to any embodiment of the present invention when executed.

[0020] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any embodiment of the present invention are implemented.

[0021] The technical solution of the embodiment of the present invention accurately locates the target operator in the target real-time stream job to be monitored in a big data cluster based on the configuration information of the real-time stream application to be monitored. During the operation of the target real-time stream job, whenever a trigger instruction is detected, the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated. When the cumulative data processing quantity is greater than or equal to a preset quantity threshold, or the cumulative processing time is greater than or equal to a preset time threshold, the current monitoring process is terminated, and a monitoring cycle is established based on the process end time and start time. Based on the cumulative data processing quantity and cumulative processing time during the monitoring cycle, the number of transactions per second and the latency time within the monitoring cycle are calculated and stored in a distributed cache, and a new trigger instruction is generated. Based on the number of transactions per second and latency times during different monitoring cycles stored in the distributed cache, a transaction number per second performance curve and a latency performance curve are obtained and displayed, thereby achieving an intuitive display of the real-time stream application performance. By writing data into the distributed cache, centralized data management is achieved, solving the data aggregation problem caused by changes in the Internet protocol and port of the monitoring address after the real-time stream application is restarted, reducing the operation risk of the real-time stream application caused by random address changes, and improving the stability and reliability of real-time stream application status monitoring.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a method for monitoring a real-time streaming application according to a first embodiment of the present invention;

[0025] Figure 2 is a flowchart of another method for monitoring real-time streaming applications provided according to embodiment 2 of the present invention;

[0026] Figure 3 is a schematic diagram of a real-time streaming application monitoring framework applicable to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of a process for calculating the number of transactions per second and latency data applicable to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of job survival monitoring applicable to an embodiment of the present invention;

[0029] Figure 6 2 is a schematic structural diagram of a monitoring device for real-time streaming applications according to a third embodiment of the present invention;

[0030] Figure 7 The figure is a schematic structural diagram of an electronic device for implementing the method for monitoring real-time streaming applications according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Example 1

[0034] Figure 1 This is a flow chart of a method for monitoring a real-time streaming application provided in the first embodiment of the present invention. This embodiment is applicable to the case of real-time monitoring and displaying the performance status of a real-time streaming application. The method can be executed by a monitoring device for a real-time streaming application. The monitoring device for a real-time streaming application can be implemented in the form of hardware and / or software and can generally be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110 : Locate, in a big data cluster, a target operator in at least one target real-time stream job to be monitored according to configuration information of the real-time stream application to be monitored.

[0036] In the embodiment of the present invention, real-time streaming can be specifically understood as: data is collected one by one in real time. Operator can be specifically understood as: a specific operation implementation represented in a real-time streaming job. Big data cluster can be specifically understood as: a system composed of multiple computers for processing and storing large-scale data, and real-time streaming applications run on the big data cluster. Each big data cluster includes at least one job, and each job includes at least one operator. The configuration information of the real-time streaming application to be monitored can be specifically understood as: a configuration file or data containing monitoring targets (such as job numbers and operator numbers, etc.) and rules, which specifies the jobs and operators to be monitored, as well as the specific monitoring method.

[0037] Specifically, in a big data cluster, multiple real-time streaming applications typically run. Each application contains multiple jobs, and each job consists of multiple operators. To effectively monitor the performance of these applications, it is necessary to locate the operators in the jobs to be monitored in the big data cluster based on the configuration information of the real-time streaming application to be monitored. In other words, it is necessary to locate the target operators in the target real-time streaming job to be monitored.

[0038] S120 . During the execution of the target real-time streaming job, whenever a trigger instruction is detected, the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated.

[0039] In an embodiment of the present invention, the trigger instruction can be specifically understood as: when the system detects that a preset rule (which can be pre-stored in the real-time stream application configuration information) is satisfied, an instruction to start the next round of monitoring. The cumulative data processing quantity can be specifically understood as: from the start time of this round of monitoring to the current time, the total number of data that the operator has processed (an accumulator can be initialized at the initial time, and each time a piece of data is processed, the accumulation is performed to obtain the cumulative data processing quantity). The cumulative processing time can be specifically understood as: from the start time of this round of monitoring to the current time, the total time used by the operator to process the data (the current time and the corresponding start time can be obtained, and the result can be calculated by subtracting the start time from the current time).

[0040] Specifically, during the operation of the target real-time streaming job, whenever a trigger instruction is detected, the moment the instruction is generated is recorded as the start time of this round of monitoring. Starting from the start time, the cumulative amount of data processed by the target operator and the total time spent by the target operator to process the data are continuously monitored.

[0041] S130: When the cumulative data processing quantity of the target operator is greater than or equal to the preset quantity threshold, or the cumulative processing time of the target operator is greater than or equal to the preset time threshold, the monitoring process is terminated, and a monitoring cycle is constructed according to the process end time and start time.

[0042] Specifically, during the monitoring period, the system monitors the cumulative amount of data processed and the cumulative processing time of the target operator in real time. When the cumulative amount is greater than or equal to the preset amount threshold, or the cumulative processing time is greater than or equal to the preset time threshold (these two thresholds are pre-stored in the real-time stream application configuration information), the monitoring process is terminated, and the current time is recorded as the end time of the monitoring process. The monitoring period corresponding to the time period between the start time and the end time of the monitoring process is used as the label of the monitoring data (cumulative data processing amount and cumulative processing time) to distinguish it from the monitoring data generated subsequently.

[0043] S140. Calculate the number of transactions per second and latency in the monitoring period based on the cumulative data processing quantity and cumulative processing time of the target operator in the monitoring period, store them in the distributed cache, and trigger the generation of a new trigger instruction.

[0044] In an embodiment of the present invention, the number of transactions per second (TPS) can be specifically understood as: a unit of measurement for software test results, that is, the result of dividing the total number of transaction processing by the transaction processing time. The delay time can be specifically understood as: the total time spent by the current operator to actually process data. In general, it can be considered that the cumulative processing time is numerically equal to the delay time. The delay time can be specifically divided into two cases: when the cumulative processing time reaches a preset time threshold, the delay time is the time window size corresponding to the preset time threshold; when the cumulative data processing quantity reaches or exceeds the preset quantity threshold, the delay time is the actual time spent on processing the data volume corresponding to the quantity threshold.

[0045] It can be understood that TPS calculation supports two strategies based on the cumulative data processing quantity and processing time (corresponding to the preset rules mentioned above), that is, the TPS data is calculated by dividing the total amount of processed data by the time interval every time a certain amount of data is processed or a certain time interval is used.

[0046] Specifically, the number of transactions per second (TPS) during the monitoring period is calculated by dividing the cumulative number of data processed by the cumulative processing time. The latency is calculated by the cumulative processing time. The calculated TPS and latency are stored in the distributed cache for subsequent query analysis. After the current monitoring period ends, the system generates a new trigger instruction to start the next monitoring period, ensuring continuous monitoring.

[0047] S150 , based on the number of transactions per second and latency of the target operator stored in the distributed cache in different monitoring cycles, obtain a transaction number per second performance curve and a latency performance curve of the target operator for display.

[0048] Specifically, based on the TPS and latency of the target operator in different monitoring cycles stored in the distributed cache, a visualization tool can be used to draw the operator's TPS performance curve (time is the horizontal axis, TPS is the vertical axis, and the TPS value of each monitoring cycle is plotted) and latency performance curve (time is the horizontal axis, latency is the vertical axis, and the latency value of each monitoring cycle is plotted) to intuitively display its performance changes.

[0049] Optionally, based on the above embodiments, a scheduled task may be set to periodically extract data from the distributed cache and update the chart to ensure that the displayed performance curve is up to date.

[0050] Optionally, based on the above embodiments, in scenarios where real-time monitoring is required, real-time updating of the chart can be achieved, that is, whenever data of a new monitoring period is generated, the corresponding curve is immediately updated.

[0051] Optionally, based on the above embodiments, exception rules are set in the monitoring rules set by the system. For example, when the rate of change of the slope of the TPS curve or the slope of the delay curve exceeds a preset threshold, it means that the performance fluctuates violently, triggering an exception prompt to ensure that the operation and maintenance personnel can respond in time.

[0052] The technical solution of the embodiment of the present invention accurately locates the target operator in the target real-time stream job to be monitored in a big data cluster based on the configuration information of the real-time stream application to be monitored. During the operation of the target real-time stream job, whenever a trigger instruction is detected, the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated. When the cumulative data processing quantity is greater than or equal to a preset quantity threshold, or the cumulative processing time is greater than or equal to a preset time threshold, the current monitoring process is terminated, and a monitoring cycle is established based on the process end time and start time. Based on the cumulative data processing quantity and cumulative processing time within the monitoring cycle, the number of transactions per second and the latency time within the monitoring cycle are calculated and stored in a distributed cache, and a new trigger instruction is generated. Based on the number of transactions per second and latency times within different monitoring cycles stored in the distributed cache, a transaction number per second performance curve and a latency performance curve are obtained and displayed, thereby achieving an intuitive display of the performance of the real-time stream application. By writing data into the distributed cache, centralized data management is achieved, solving the data aggregation problem caused by changes in the Internet protocol and port of the monitoring address after the real-time stream application is restarted, reducing the operation risk of the real-time stream application caused by random address changes, and improving the stability and reliability of real-time stream application status monitoring.

[0053] Furthermore, based on the above embodiments, each time a trigger instruction is detected, the following steps may be further included:

[0054] Reset the cumulative data processing quantity counter of the target operator, reset the cumulative processing time timer of the target operator, and reset the cumulative processing time of the target operator;

[0055] Accordingly, real-time monitoring of the target operator's cumulative data processing quantity and cumulative processing time may include:

[0056] Before the target operator starts processing a target data, the current timestamp is added to the target data;

[0057] When it is determined that the target operator has completed processing the target data, the cumulative data processing quantity counter of the target operator is updated to monitor the cumulative data processing quantity of the target operator in real time; and when it is determined that the target operator has completed processing the target data, the current timing value is obtained from the cumulative processing time timer of the target operator, and the time difference between the current timing value and the timestamp in the target data is calculated;

[0058] According to the calculated time difference, the cumulative processing time of the target operator is updated to monitor the calculated cumulative processing time in real time.

[0059] In the embodiments of the present invention, the cumulative data processing quantity counter can be specifically understood as a counter for counting the amount of data processed by the target operator starting from the time the trigger instruction is generated. The cumulative processing time timer can be specifically understood as a timer for counting the total time spent by the target operator processing data starting from the time the trigger instruction is generated.

[0060] Specifically, during the operation of real-time streaming jobs, whenever the system detects a new trigger instruction, the cumulative data processing quantity counter of the target operator is reset to 0, the cumulative processing time timer of the target operator is reset to 0, and the cumulative processing time of the target operator is reset to 0, ensuring that at the beginning of a new monitoring cycle, the cumulative data processing quantity and cumulative processing time are counted from 0.

[0061] Accordingly, before the target operator begins processing the target data, the system obtains the current timestamp and associates it with the data, marking the start time of data processing. After the operator completes data processing, the system increments the cumulative data processing counter by 1, reflecting the total amount of data processed by the operator in real time. After data processing is complete, the system obtains the current value from the cumulative processing time counter and subtracts it from the start timestamp of the current data to determine the time spent processing the data. The system then adds this calculated time difference to the target operator's cumulative processing time, updating the total time it took the operator to process all data in real time.

[0062] By resetting the counters and timers at the beginning of each new monitoring cycle, we ensure that each monitoring cycle starts from zero, accurately counting the amount of data processed and the processing time within that cycle. This prevents data from the previous monitoring cycle from influencing the new monitoring cycle, ensuring the independence and accuracy of the data for each cycle. By accumulating the processing time of each data item, we can accurately determine the processing details of each data item, avoiding statistical errors caused by fluctuations in operator processing tasks (such as idle or waiting operator tasks), accurately reflecting the actual time consumed by data processing, and improving the system's monitoring accuracy of cumulative processing time and cumulative data processed.

[0063] Optionally, based on the above embodiments, the number of transactions per second and latency in the monitoring period are calculated based on the cumulative amount of data processed and the cumulative processing time of the target operator in the monitoring period and stored in the distributed cache, which may include:

[0064] Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second during the monitoring period is calculated and stored in the distributed cache;

[0065] Obtain the current timestamp, calculate the target operator's latency based on the timestamp of the first target data processed within the target operator's monitoring period and the current timestamp, and store the latency in the distributed cache.

[0066] Specifically, based on the cumulative amount of data processed and the cumulative processing time of the target operator during the monitoring cycle, the cumulative amount of data processed is divided by the cumulative processing time to calculate the number of transactions per second during the monitoring cycle and store it in the distributed cache. At the end of the monitoring cycle, the current timestamp is obtained, as well as the timestamp added to the data when the target operator processes the first data during the monitoring cycle (that is, the time when the target operator starts processing data in this round of monitoring cycle). The first data timestamp is subtracted from the current timestamp to obtain the target operator's delay time (that is, the delay time of the current operator in this round of monitoring cycle), and stored in the distributed cache. The TPS and delay time are calculated using the timestamp added to the data, providing data support for performance evaluation and improving the stability and reliability of real-time stream application status monitoring.

[0067] Optionally, based on the above embodiments, a real-time stream job may include multiple operator groups, each of which may include: a data consumption operator, zero or at least one data conversion operator, and a data production operator connected in series;

[0068] Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second and latency during the monitoring period are calculated and stored in the distributed cache. This also includes:

[0069] According to the operator type of the target operator, an inter-operator delay time between the target operator and at least one upstream operator of the target operator in the operator group to which the target operator belongs is calculated.

[0070] In an embodiment of the present invention, a data consumption operator can be specifically understood as: an operator that reads a data stream from a data source (such as a distributed stream processing platform, a message queue system or a database, etc.), passes the acquired data stream to a data conversion operator for processing, or directly passes it to a data production operator for output. A data conversion operator can be specifically understood as: an operator that receives a data stream from a data consumption operator or other upstream data conversion operators, and processes it (such as cleaning, conversion and aggregation operations), and passes the processed data stream to a data production operator for output or other downstream data conversion operators for further processing. A data production operator can be specifically understood as an operator that receives a data stream from a data conversion operator or a data consumption operator, and outputs the data stream to a target system (such as a database, a file system or a message queue, etc.). An upstream operator can be specifically understood as an operator that directly or indirectly provides data to a target operator during the processing of a real-time stream application job.

[0071] It is understandable that the real-time stream application configuration information may include: a job number, an operator number, and a system name or number (such as the name or number of a data source and a target system).

[0072] Specifically, a real-time streaming job can include multiple operator groups, each corresponding to a connection method between operators. Each operator group can include: a data consumption operator responsible for reading data from the data source; a data production operator responsible for outputting data to the target system. Optionally, it can also include at least one data conversion operator responsible for converting and processing data.

[0073] It can be understood that the calculation of the inter-operator delay time can cover the entire process from the upstream operator receiving data (especially, from the data consumption operator receiving data) to the target operator completing data processing in the current monitoring cycle (especially, the data production operator completing data output to the target system in the current monitoring cycle). This calculation process depends on the linkage and collaboration between upstream and downstream operators.

[0074] When data enters each operator, a corresponding timestamp is added, indicating the time when the data began to be processed by that operator. Therefore, at the end of the monitoring cycle, the data processed by the target operator contains timestamps added by the target operator and each upstream operator, corresponding to the time when the target operator and each upstream operator began processing the data. In particular, based on the current timestamp at the end of the monitoring cycle and the upstream timestamp added by the upstream operator corresponding to the target operator in the first data processed by the target operator (i.e., the time when the upstream operator corresponding to the target operator began processing data in this round of monitoring cycle), the inter-operator latency can be calculated.

[0075] Specifically, based on the configuration information (including the number of the currently monitored operator and the number of the upstream operator used to calculate the inter-operator delay), for the target operator, determine at least one corresponding upstream operator in the operator group to which it belongs. At the end of the monitoring cycle, obtain the current timestamp and the upstream timestamp added by the upstream operator corresponding to the target operator in the first data processed by the target operator during the monitoring cycle (that is, the time when the upstream operator corresponding to the target operator started processing data in this monitoring cycle). Subtract the upstream timestamp from the current timestamp (that is, the time when the target operator ends processing in this monitoring cycle minus the time when the upstream operator starts processing data in this monitoring cycle) to calculate the inter-operator delay between the target operator and the upstream operator and store it in the distributed cache.

[0076] In a specific example, when the target operator is a data consumption operator, since it is located at the beginning of the operation process and there is no upstream operator, the delay time between operators cannot be calculated, but the delay time of the target operator itself, that is, the delay time of the target operator, can be calculated.

[0077] When the target operator is a data conversion operator, its upstream operator can be other data conversion operators or data consumption operators. Therefore, in addition to calculating the target operator's latency (the target operator's own latency), you can also calculate the inter-operator latency between the target operator and other data conversion operators, or the inter-operator latency between the target operator and data consumption operators.

[0078] When the target operator is a data production operator, since it is located at the end of the job flow, its upstream operator can be a data conversion operator or a data consumption operator. Therefore, in addition to calculating the target operator's latency (the target operator's own latency), the inter-operator latency between the target operator and the data conversion operator can also be calculated, or the inter-operator latency between the target operator and the data consumption operator can be calculated.

[0079] Typically, data consuming operators receive input from external data sources, undergo a series of processing steps, and finally output the processed results to data producing operators. Therefore, the latency from a data consuming operator to a data producing operator encompasses the data processing process of all operators in the entire job, including intermediate links such as data conversion operators. The inter-operator latency between data producing and consuming operators represents the overall inter-operator latency for the entire job.

[0080] By calculating TPS and the latency of the target operator, the performance of individual operators in real-time streaming applications can be monitored. By calculating and monitoring inter-operator latency, the data processing flow of each operator group in a real-time streaming job can be comprehensively monitored, understanding the data processing efficiency and latency at each stage, thereby enabling refined management of the entire data processing process. In particular, by monitoring the overall latency between operators throughout the entire job, the overall performance of the real-time streaming application can be monitored, improving the stability and reliability of real-time streaming application status monitoring.

[0081] Optionally, based on the above embodiments, the data structure of the distributed cache includes: a data subdomain and an extension subdomain;

[0082] The data subdomain is used to store data information; the extension subdomain is used to save the timestamp information corresponding to the processed data information.

[0083] Specifically, considering the inconsistencies in the data structures of different raw data, in order to facilitate unified processing, a universal data structure is first defined. This data structure contains two subdomains (each subdomain stores data in the form of key-value pairs), namely the data subdomain and the extension subdomain. The data subdomain is used to store data information and encapsulate the real-time collected data into the data subdomain; the extension subdomain is used to store the specific timestamp information corresponding to the processed data information (such as the timestamp added when the target operator starts processing this data), which can be used to calculate latency and TPS. The extension subdomain can expand the storage content according to actual business needs. Through the collaborative work of the data subdomain and the extension subdomain, the data subdomain provides business data, and the extension subdomain provides the time context of data processing, which improves the monitoring ability of data processing, enables the system to effectively manage and optimize data streams, and improves the stability and reliability of real-time stream application status monitoring.

[0084] Optionally, based on the above embodiments, the real-time streaming application configuration information may include: a task number;

[0085] Accordingly, based on the number of transactions per second and latency of the target operator stored in the distributed cache in different monitoring cycles, a transaction number per second performance curve and a latency performance curve of the target operator are obtained, which may include:

[0086] The total number of transactions per second of the target operator for all tasks of the same real-time stream job within each monitoring period is calculated as the total number of transactions per second of the target operator.

[0087] Draw the target operator's transaction per second performance curve based on the total number of transactions per second corresponding to each monitoring period;

[0088] The maximum latency of all tasks of the same target operator in the same real-time stream job within each monitoring period is taken as the total latency of the target operator.

[0089] Draw a latency performance curve based on the total latency of the target operator corresponding to each monitoring period.

[0090] Specifically, the real-time stream application configuration information may include a task number, which is used to uniquely identify a specific task in the real-time stream job, and to accurately locate and monitor the operator performance of the corresponding task in the distributed cache.

[0091] Generally speaking, in a distributed stream processing framework, operator parallelism can be understood as the ability of multiple task instances of the same operator to run simultaneously. Therefore, the task number can also be understood as the parallelism number.

[0092] Specifically, during each monitoring cycle, the number of transactions per second (TSP) for all tasks of the same target operator in the same real-time stream job is summed to obtain the target operator's total number of transactions per second. Based on the TPS data points for each monitoring cycle, a TPS performance curve is plotted for the target operator, showing the changes in the target operator's data throughput over time.

[0093] Because maximum latency represents the longest amount of time spent processing data within a given cycle, it significantly impacts system performance. Therefore, during each monitoring cycle, the maximum latency of all tasks within the same real-time stream job and the same target operator is taken as the total latency for that target operator. Based on the total latency data points from each monitoring cycle, a latency performance curve for the target operator is plotted, reflecting its latency variations over time.

[0094] The TPS of a single operator in a real-time streaming application is summed up to form a summary indicator and output in the form of a curve. At the same time, the data processing delay between different operators is provided and output in the form of a curve. This increases the monitoring indicators between operators, makes the display of the performance of the entire real-time streaming application more intuitive, and improves the stability and reliability of real-time streaming application status monitoring.

[0095] Optionally, based on the above embodiments, the data structure of the distributed cache can be further refined into two parts: a data structure of the distributed cache for storing TPS and latency time data and a data structure of the distributed cache for storing TPS performance curves and latency performance curves.

[0096] The data structure for storing TPS and latency data in a distributed cache is designed as follows: the key of the distributed cache is "stat:tps:delay" and is used to store data related to performance metrics. "stat" indicates the start of a specific operation associated with this parameter group, such as collecting data for these metrics or starting a process to analyze and monitor these metrics. "tps" represents transactions per second, and "delay" represents latency. The fields describe the context of the monitoring metric and uniquely identify a target operator, consisting of "job number:task number:operator number." The job number identifies a specific job in a real-time streaming application, the task number identifies a specific task within a job, and the operator number identifies the specific target operator within a task. The value includes data for specific monitoring metrics: for consuming operators, this includes the source system (the name or number of the system where the data originates) and the TPS of the consuming operator; for producing operators, this includes the target system (the name or number of the system to which the data is sent), the target TPS (the TPS of the producing operator), and the processing time (the time consumed by the job, i.e., latency). The conversion operator includes: source operator (name or number of the operator where the data comes from), data conversion operator TPS and target operator (name or number of the target operator where the data is sent).

[0097] That is, the data structure of the distributed cache for storing TPS and delay time data can adopt the hash data type, the cache key is fixed as: stat:tps:delay, and the field uses the real-time stream job number, operator parallelism number (i.e. task number) and operator number to uniquely identify a certain operator parallelism data. Each operator is responsible for recording its own job number, operator parallelism number and operator number and other information, and calculates its own TPS and processing delay information when the preset conditions are met and writes it into the distributed cache storage.

[0098] The data structure design of the distributed cache for storing TPS performance curves and latency performance curves is as follows:

[0099] The key of the distributed cache uses "tps:linechart:job number" as the key identifier, which uniquely identifies the performance monitoring data of a specific job. "linechart" indicates that the data is visualized using a line chart (other formats can be replaced as needed). The value includes: a timestamp, which records the time when the corresponding data was generated (corresponding to the monitoring period); an operator's TPS, which records the operator's TPS value; and a processing time, which records the data latency in real-time streaming applications (corresponding to the latency of the monitoring period).

[0100] Specifically, the data structure of the distributed cache used to store TPS and latency performance curves can be designed using a list structure, with each real-time streaming job having a key and a value containing the timestamp, operator TPS, and latency data corresponding to the monitoring period. By further refining the distributed cache structure, data processing monitoring capabilities are enhanced, enabling the system to effectively manage and optimize data flows and improving the stability and reliability of real-time streaming application status monitoring.

[0101] Example 2

[0102] Figure 2 A flowchart of another monitoring method for real-time stream applications provided in Example 2 of the present invention. This embodiment is a refinement of the monitoring method for real-time stream applications in the above embodiment, and specifically may include: while monitoring the cumulative data processing quantity and cumulative processing time of the target operator in real time, it may also include: monitoring the survival status and job restart count of at least one real-time stream job in real time according to the real-time stream application configuration information; when the real-time stream job status is alive and the restart count is greater than 0, the restart count is reset to 0; when the real-time stream job status is not alive and the restart count is greater than or equal to the set restart threshold, a prompt is issued that the job cannot be automatically pulled up and restored; when the real-time stream job status is not alive and the restart count is less than the set restart threshold, the current job is automatically submitted and the restart count is updated.

[0103] Correspondingly, such as Figure 2 As shown, the method includes:

[0104] S210: Locate, in the big data cluster, a target operator in at least one target real-time stream job to be monitored according to the real-time stream application configuration information to be monitored.

[0105] S220: During the execution of the target real-time streaming job, whenever a trigger instruction is detected, the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated.

[0106] S230: When the cumulative data processing quantity of the target operator is greater than or equal to the preset quantity threshold, or the cumulative processing time of the target operator is greater than or equal to the preset time threshold, the monitoring process is terminated, and a monitoring cycle is constructed according to the process end time and start time.

[0107] S240: Calculate the number of transactions per second and latency time during the monitoring period based on the cumulative data processing quantity and cumulative processing time of the target operator during the monitoring period, store them in the distributed cache, and trigger the generation of a new trigger instruction.

[0108] S250: According to the number of transactions per second and the latency of the target operator stored in the distributed cache in different monitoring cycles, a transaction number per second performance curve and a latency performance curve of the target operator are obtained and displayed.

[0109] S260. While monitoring the cumulative data processing amount and cumulative processing time of the target operator in real time, monitor the survival status and job restart count of at least one real-time stream job in real time according to the real-time stream application configuration information. When the real-time stream job status is alive and the restart count is greater than 0, reset the restart count to 0.

[0110] S270: When the real-time streaming job status is not alive and the restart times are greater than or equal to the set restart threshold, a prompt is issued that the job cannot be automatically pulled up and restored.

[0111] S280: When the real-time streaming job status is not alive and the restart count is less than the set restart threshold, the current job is automatically submitted and the restart count is updated.

[0112] Specifically, while the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, the survival status and job restart count of the matching real-time streaming job are monitored in real time based on the real-time streaming application configuration information (including the number of the real-time streaming job that needs to be monitored). When the real-time streaming job is in a survival state, but the restart count is greater than 0, the restart count is reset to 0, and the restart count is cleared to ensure the accuracy of the count, which is convenient for subsequent monitoring and automatic re-provisioning of the job. When the real-time streaming job is in a non-survival state and the restart count reaches or exceeds the preset threshold, a prompt is issued that the job cannot be automatically pulled up and restored, and timely intervention is made to avoid long-term job abnormalities. When the real-time streaming job is in a non-survival state but the restart count does not reach the preset threshold, the current job is automatically submitted and the restart count is updated (1 is added to the current updated restart count).

[0113] The technical solution of the embodiment of the present invention locates the target operator based on configuration information; during operation, whenever a trigger instruction is detected, the cumulative data processing quantity and cumulative processing time are monitored in real time, starting from the time when the trigger instruction is generated; when the cumulative data processing quantity is greater than or equal to the quantity threshold, or the cumulative processing time is greater than or equal to the time threshold, the monitoring process is terminated, and a monitoring cycle is constructed based on the end time and start time. The number of transactions per second and the latency time within the monitoring cycle are calculated and stored in the distributed cache, and a new trigger instruction is generated; based on the number of transactions per second and the latency time in the cache, a transaction number per second performance curve and a latency performance curve are obtained for display, thereby achieving an intuitive display of the performance of the real-time streaming application. By writing data into the distributed cache, the operation risk of the real-time streaming application caused by random address changes is reduced. By simultaneously tracking the processing progress of the target operator and the running status of the job, comprehensive monitoring of the real-time streaming application is achieved. When a job survives, the restart count is automatically reset, eliminating the risk of false alarms. When a job is abnormal and the restart count reaches the limit, a prompt is issued in a timely manner to ensure system reliability. When a job does not survive but the restart count does not reach the threshold, the job is automatically restarted, improving the system's automation level and availability. It can not only effectively manage the job restart process, but also restore the application when anomalies are caused by external factors, ensuring the continuous and stable operation of real-time streaming applications, and improving the stability and reliability of real-time streaming application status monitoring.

[0114] For ease of understanding, the specific application scenarios to which the above-mentioned embodiments of the invention are applicable are described. With the continuous development of Internet technology, the data generated by applications has increased massively. In order to extract valuable information from massive data, big data technology and real-time computing frameworks have emerged to realize the storage and analysis of massive data. As a key subject of data processing, the performance of real-time streaming applications directly determines the efficiency and effectiveness of data processing. In the prior art, the built-in monitoring function of real-time streaming applications is usually used to view the relevant data indicators of operators, and most of them focus on file-related monitoring and analysis. There are problems such as the inability to view the processing delay of job data, the change of the job running monitoring address after restart, resulting in monitoring failure, and the inability to automatically pull up in the event of an abnormal job crash. In order to solve the above problems, the embodiments of the present invention propose a monitoring method for real-time streaming applications, Figure 3 Schematic diagram of a real-time stream application monitoring framework applicable to an embodiment of the present invention. Figure 3As shown, the architecture uses a configuration center to store configuration data (can store multiple real-time stream application configuration information, such as N configuration information with labels 1 to N), uses a distributed cache cluster to store the TPS and delay data generated by the real-time stream job operator calculation, and stores the TPS performance curve and delay performance curve written by the timing dotting application. The timing monitoring application is used to monitor the survival status of the real-time stream job, and the detection cycle configuration is hosted in the configuration center, which can be flexibly adjusted according to actual needs; the timing dotting application summarizes the multi-parallelism data of the same operator of the real-time stream job, calculates the total TPS and maximum delay of the operator, and writes them into the distributed cache storage at regular intervals, and can generate corresponding data change curves based on the data rendering and write them into the distributed cache. Accordingly, the method can specifically include:

[0115] 1. Real-time streaming operator performance monitoring

[0116] Taking into account the inconsistency of different raw data structures, in order to facilitate unified processing, a universal data structure is first defined. This data structure contains two subdomains (each subdomain stores data in the form of key-value pairs), namely the data subdomain and the extension subdomain. The data subdomain is used to store data information and encapsulate the real-time collected data into the data subdomain; the extension subdomain is used to store specific timestamp information, which can be used to calculate latency and TPS, and can be expanded according to actual business needs.

[0117] Existing real-time streaming applications generally include three types of operators: data consumption, data conversion, and data production. Taking the TPS of data consumption operators, data production operators, and data processing latency calculation as examples, real-time streaming operator performance monitoring can specifically include:

[0118] (1) TPS and latency calculation

[0119] First, the distributed cache is divided into two areas. The data structure of the distributed cache for storing TPS and latency data is designed as follows:

[0120] Distributed cache keys use "stat:tps:delay" as key identifiers to store data related to performance metrics. "stat" indicates the start of specific operations related to this parameter group, such as collecting data for these metrics or starting a process for analyzing and monitoring these metrics. "tps" represents transactions per second, and "delay" represents latency. Fields describe the context of the monitoring metrics, uniquely identifying a specific operator (equivalent to the target operator for real-time monitoring described above). These fields consist of two main components: "job number:task number:consumer operator number" and "job number:task number:producer operator number." The job number identifies a specific job in a real-time streaming application, the task number identifies a specific task within a job, the consumer operator number identifies a specific consumer operator within a task, and the producer operator number identifies a specific producer operator within a task. Value: Contains data on specific monitoring indicators. For consumer operators, this includes the source system (the name of the system where the data originates) and the TPS of the consumer operator. For producer operators, this includes the target system (the name of the system to which the data is sent), target TPS (the TPS of the producer operator), and time consumed (the time consumed for job processing).

[0121] That is, the data structure of the distributed cache for storing TPS and delay time data uses a hash data type, the cache key is fixed to: stat:tps:delay, and the field uses the real-time stream job number, operator parallelism number (i.e., task number), and operator number to uniquely identify the parallelism data of a certain operator. Each operator is responsible for recording its own job number, operator parallelism number, operator number and other information, and calculates its own TPS and processing delay information when the preset conditions are met and writes it into the distributed cache storage.

[0122] Figure 4 FIG. 1 is a schematic diagram of a calculation process of transaction counts per second and latency data applicable to an embodiment of the present invention. Figure 4As shown, for a data consumption operator: For the first monitoring, before starting data consumption, a counter (equivalent to the cumulative data processing counter mentioned above) and a timestamp (equivalent to the cumulative processing time of the target operator mentioned above) are initialized. Based on the configuration information, data is retrieved from the target data source, and the counter is updated simultaneously. The current timestamp is obtained to calculate the size of the time window (preset time threshold) (i.e., the time to trigger the calculation of its own TPS and processing delay information and write it to the distributed cache can be obtained by adding the current timestamp to the allowed operator data processing time, which serves as the preset time threshold). The current data is then timestamped. The operator monitors whether the counter has reached the preset threshold or whether the time window has reached the preset size (equivalent to checking whether the cumulative data processing of the target operator is greater than or equal to the preset number threshold, or whether the cumulative processing time of the target operator is greater than the preset time threshold). If so, the operator's TPS is calculated, the result is output to the distributed cache, the counter and timestamp are reset, the next round of monitoring begins, and finally the acquired data is output. If not, the acquired data is directly output.

[0123] For the data production operator: For the first monitoring, before starting to produce data, initialize the counter and timestamp. Obtain the data output by the data consumption operator (data conversion operators are not considered here, and the data consumption operator and data production operator are directly connected), update the counter, obtain the current timestamp to calculate the size of the time window, and timestamp the current data. Monitor whether the counter reaches the preset threshold or whether the time window reaches the preset size. If so, calculate the TPS of the current operator, output the result to the distributed cache, reset the counter and timestamp, and start the next round of monitoring. Then, before the data is output, obtain the current timestamp, calculate the entire processing delay from data consumption to data output (which can be calculated by subtracting the timestamp added by the consumer operator in the first processed data in the current monitoring cycle from the current timestamp), and output the delay to the distributed cache. Finally, output the data. If not, output the data directly.

[0124] TPS calculation supports two strategies: counter-based and time window-based. Specifically, TPS is calculated by dividing the total amount of processed data by the time interval after a certain amount of data is processed or after a certain time interval. Latency calculation relies on the linkage between upstream and downstream operators. Data consumption operators store timestamp information in the data extension domain structure. Data production operators calculate the corresponding data latency based on the current time and the timestamp information stored by the data consumption operators.

[0125] (2) Generation of TPS performance curve and delay performance curve

[0126] The data structure design of the distributed cache for storing TPS performance curves and latency performance curves is as follows:

[0127] The key of the distributed cache uses "tps:linechart:job number" as the key identifier, which uniquely identifies the performance monitoring data of a specific job. "linechart" indicates that the data is visualized using a line chart. The value includes: timestamp, which records the time point of the data (corresponding to the monitoring period); source TPS, which records the TPS value of the data consuming operator (corresponding to the TPS value of the consuming operator during the monitoring period); target TPS, which records the TPS value of the data producing operator; and processing time, which records the data latency in real-time streaming applications (corresponding to the latency of the monitoring period).

[0128] That is, the data structure of the distributed cache for storing TPS performance curves and latency performance curves adopts a list structure design. Each real-time stream job has a key, and the value includes the scheduled check time (corresponding to the monitoring period), the TPS of the data consumption operator, the TPS of the data production operator, and the data processing latency time data.

[0129] Among them, the scheduled application regularly obtains the TPS and latency information written by each job operator from the distributed cache. The TPS and latency calculation method for the same job and the same operator with different parallelism is: take the sum of the TPS under all parallelism as the total TPS of the operator, and take the maximum latency under all parallelism as the total latency of the operator.

[0130] The processed data is stored in the distributed cache at regular intervals. The length of the distributed cache list is determined according to actual business needs. When new data is written to the list, the old data can be eliminated to keep the list length stable.

[0131] 2. Real-time streaming job survival monitoring

[0132] Figure 5 FIG. 1 is a schematic diagram of job survival monitoring applicable to an embodiment of the present invention, such as Figure 5 As shown, the scheduled monitoring application queries the list of monitored real-time streaming jobs through the configuration center, and performs corresponding processing by judging the survival status of the job. When it is found that the job is not alive, it is judged whether the number of job restarts is greater than or equal to the specified threshold (such as 3 times). If it is greater than or equal to the specified threshold, it is considered that the job cannot be recovered by automatic pull-up. When the number of job restarts is less than the specified threshold, the corresponding job is submitted and the submission count is updated; if it is found that the job is alive, the number of job restarts is judged. When the number of restarts is greater than zero, the number of restarts is reset to zero. In this way, automatic restart is achieved when the job crashes abnormally.

[0133] The monitoring method for real-time stream applications proposed in an embodiment of the present invention forms a summary indicator for the TPS of a single operator of the real-time stream, which more intuitively displays the performance of the entire real-time stream application. At the same time, it provides data processing delays between different operators, increases monitoring indicators between operators, and supports the generation of monitoring indicators between more operators based on the extension domain in the data interface. By writing data of different parallelisms of different operators of the real-time stream job into a distributed cache storage, data aggregation is achieved, avoiding changes in the Internet protocol and port corresponding to the monitoring address after the real-time stream application is restarted, and reducing production and operation risks caused by random Internet protocols and ports. While monitoring the performance of the operators within the real-time stream job, the survival status of the real-time stream job itself is also monitored accordingly. In abnormal situations caused by certain external factors, the application can be automatically restored to a great extent, ensuring the stable operation of the real-time stream application.

[0134] Example 3

[0135] Figure 6 This is a structural diagram of a monitoring device for real-time streaming applications provided by the third embodiment of the present invention. Figure 6 As shown, the device includes: a positioning operator module 610, a real-time monitoring module 620, a monitoring cycle module 630, a calculation and storage module 640 and a curve display module 650, wherein:

[0136] The positioning operator module 610 is used to locate a target operator in at least one target real-time stream job to be monitored in the big data cluster according to the configuration information of the real-time stream application to be monitored;

[0137] The real-time monitoring module 620 is used to monitor the cumulative data processing amount and cumulative processing time of the target operator in real time whenever a trigger instruction is detected during the operation of the target real-time streaming job, starting from the time when the trigger instruction is generated;

[0138] The monitoring cycle module 630 is configured to terminate the monitoring process when the cumulative data processing amount of the target operator is greater than or equal to a preset number threshold, or the cumulative processing time of the target operator is greater than or equal to a preset time threshold, and to establish a monitoring cycle based on the process end time and start time;

[0139] The calculation and storage module 640 is used to calculate the number of transactions per second and the latency in the monitoring period based on the cumulative data processing quantity and cumulative processing time of the target operator in the monitoring period, store them in the distributed cache, and trigger the generation of a new trigger instruction;

[0140] The curve display module 650 is used to obtain the transaction number per second performance curve and the latency performance curve of the target operator according to the transaction number per second and latency of the target operator stored in the distributed cache in different monitoring cycles for display.

[0141] The technical solution of the embodiment of the present invention accurately locates the target operator in the target real-time stream job to be monitored in a big data cluster based on the configuration information of the real-time stream application to be monitored. During the operation of the target real-time stream job, whenever a trigger instruction is detected, the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated. When the cumulative data processing quantity is greater than or equal to a preset quantity threshold, or the cumulative processing time is greater than or equal to a preset time threshold, the current monitoring process is terminated, and a monitoring cycle is established based on the process end time and start time. Based on the cumulative data processing quantity and cumulative processing time within the monitoring cycle, the number of transactions per second and the latency time within the monitoring cycle are calculated and stored in a distributed cache, and a new trigger instruction is generated. Based on the number of transactions per second and latency times within different monitoring cycles stored in the distributed cache, a transaction number per second performance curve and a latency performance curve are obtained and displayed, thereby achieving an intuitive display of the performance of the real-time stream application. By writing data into the distributed cache, centralized data management is achieved, solving the data aggregation problem caused by changes in the Internet protocol and port of the monitoring address after the real-time stream application is restarted, reducing the operation risk of the real-time stream application caused by random address changes, and improving the stability and reliability of real-time stream application status monitoring.

[0142] Furthermore, based on the above embodiments, the monitoring device for real-time streaming applications may further include a reset module, wherein:

[0143] The reset module is used to reset the cumulative data processing quantity counter of the target operator, reset the cumulative processing time timer of the target operator, and reset the cumulative processing time of the target operator every time a trigger instruction is detected.

[0144] Accordingly, based on the above embodiments, the real-time monitoring module 620 is specifically configured to:

[0145] Before the target operator starts processing a target data, the current timestamp is added to the target data;

[0146] When it is determined that the target operator has completed processing the target data, the cumulative data processing quantity counter of the target operator is updated to monitor the cumulative data processing quantity of the target operator in real time; and when it is determined that the target operator has completed processing the target data, the current timing value is obtained from the cumulative processing time timer of the target operator, and the time difference between the current timing value and the timestamp in the target data is calculated;

[0147] According to the calculated time difference, the cumulative processing time of the target operator is updated to monitor the calculated cumulative processing time in real time.

[0148] Based on the above embodiments, the calculation storage module 640 is specifically configured to:

[0149] Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second during the monitoring period is calculated and stored in the distributed cache;

[0150] Obtain the current timestamp, calculate the target operator's latency based on the timestamp of the first target data processed within the target operator's monitoring period and the current timestamp, and store the latency in the distributed cache.

[0151] Based on the above embodiments, a real-time stream job may include multiple operator groups, and each operator group may include: a data consumption operator connected in series, 0 or at least one data conversion operator and a data production operator.

[0152] Furthermore, based on the above embodiments, the monitoring device for real-time streaming applications may further include an inter-operator delay module, wherein:

[0153] The inter-operator delay module is used to calculate the number of transactions per second and the delay time within the monitoring period based on the cumulative data processing quantity and cumulative processing time of the target operator within the monitoring period and store them in the distributed cache. At the same time, according to the operator type of the target operator, it calculates the inter-operator delay time between the target operator and at least one upstream operator of the target operator in the operator group to which the target operator belongs.

[0154] Based on the above embodiments, the data structure of the distributed cache may include: a data subdomain and an extension subdomain;

[0155] The data subdomain is used to store data information; the extension subdomain is used to save the timestamp information corresponding to the processed data information.

[0156] Based on the above embodiments, the real-time streaming application configuration information may include: a task number;

[0157] Accordingly, based on the above embodiments, the curve display module 650 is specifically configured to:

[0158] The total number of transactions per second of the target operator for all tasks of the same real-time stream job within each monitoring period is calculated as the total number of transactions per second of the target operator.

[0159] Draw the target operator's transaction per second performance curve based on the total number of transactions per second corresponding to each monitoring period;

[0160] The maximum latency of all tasks of the same target operator in the same real-time stream job within each monitoring period is taken as the total latency of the target operator.

[0161] Draw a latency performance curve based on the total latency of the target operator corresponding to each monitoring period.

[0162] Furthermore, based on the above embodiments, the monitoring device for real-time streaming applications may further include: a job monitoring module, an inability to pull up module, and an automatic submission module, wherein:

[0163] The job monitoring module is used to monitor the cumulative data processing amount and cumulative processing time of the target operator in real time. At the same time, it monitors the survival status and job restart count of at least one real-time streaming job in real time according to the real-time streaming application configuration information. When the real-time streaming job status is alive and the restart count is greater than 0, the restart count is reset to 0;

[0164] The Unable to Restart module is used to issue a prompt indicating that the job cannot be automatically restarted when the real-time streaming job status is not alive and the number of restarts is greater than or equal to the set restart threshold;

[0165] The automatic submission module is used to automatically submit the current job and update the restart count when the real-time streaming job status is not alive and the restart count is less than the set restart threshold.

[0166] The monitoring device for real-time streaming applications provided by the embodiment of the present invention can execute the monitoring method for real-time streaming applications provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0167] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0168] Example 4

[0169] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0170] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0171] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0172] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the monitoring method for real-time streaming applications, namely:

[0173] Locating, in the big data cluster, a target operator in at least one target real-time stream job to be monitored according to the real-time stream application configuration information to be monitored;

[0174] During the execution of the target real-time streaming job, whenever a trigger instruction is detected, the cumulative data processing amount and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated;

[0175] When the cumulative data processing volume of the target operator is greater than or equal to the preset volume threshold, or the cumulative processing time of the target operator is greater than or equal to the preset time threshold, the monitoring process ends and a monitoring cycle is established based on the process end time and start time.

[0176] Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second and latency during the monitoring period are calculated and stored in the distributed cache, and a new trigger instruction is generated;

[0177] Based on the number of transactions per second and latency of the target operator stored in the distributed cache in different monitoring cycles, the target operator's transaction number per second performance curve and latency performance curve are obtained for display.

[0178] In some embodiments, the method for monitoring a real-time streaming application can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for monitoring a real-time streaming application described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for monitoring a real-time streaming application in any other appropriate manner (e.g., by means of firmware).

[0179] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0184] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0185] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0186] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for monitoring a real-time streaming application, characterized in that: include: Locating, in the big data cluster, a target operator in at least one target real-time stream job to be monitored according to the real-time stream application configuration information to be monitored; During the execution of the target real-time streaming job, whenever a trigger instruction is detected, the cumulative data processing amount and cumulative processing time of the target operator are monitored in real time, starting from the time when the trigger instruction is generated; When the cumulative data processing volume of the target operator is greater than or equal to the preset volume threshold, or the cumulative processing time of the target operator is greater than or equal to the preset time threshold, the monitoring process ends and a monitoring cycle is established based on the process end time and start time. Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second and latency during the monitoring period are calculated and stored in the distributed cache, and a new trigger instruction is generated; Based on the number of transactions per second and latency of the target operator stored in the distributed cache in different monitoring cycles, the target operator's transaction number per second performance curve and latency performance curve are obtained for display.

2. The method according to claim 1, characterized in that Whenever a trigger instruction is detected, it also includes: Reset the cumulative data processing quantity counter of the target operator, reset the cumulative processing time timer of the target operator, and reset the cumulative processing time of the target operator; Accordingly, the cumulative data processing quantity and cumulative processing time of the target operator are monitored in real time, including: Before the target operator starts processing a target data, the current timestamp is added to the target data; When it is determined that the target operator has completed processing the target data, updating the cumulative data processing quantity counter of the target operator to monitor the cumulative data processing quantity of the target operator in real time; and When it is determined that the target operator has completed processing the target data, the current timing value is obtained from the cumulative processing time timer of the target operator, and the time difference between the current timing value and the timestamp in the target data is calculated; According to the calculated time difference, the cumulative processing time of the target operator is updated to monitor the calculated cumulative processing time in real time.

3. The method according to claim 2, characterized in that Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second and latency during the monitoring period are calculated and stored in the distributed cache, including: Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second during the monitoring period is calculated and stored in the distributed cache; Obtain the current timestamp, calculate the target operator's latency based on the timestamp of the first target data processed within the target operator's monitoring period and the current timestamp, and store the latency in the distributed cache.

4. The method according to claim 2, characterized in that A real-time stream job includes multiple operator groups. Each operator group includes: a sequentially connected data consumption operator, zero or at least one data conversion operator, and a data production operator. Based on the cumulative data processing volume and cumulative processing time of the target operator during the monitoring period, the number of transactions per second and latency during the monitoring period are calculated and stored in the distributed cache. This also includes: According to the operator type of the target operator, an inter-operator delay time between the target operator and at least one upstream operator of the target operator in the operator group to which the target operator belongs is calculated.

5. The method according to claim 1, characterized in that The data structure of the distributed cache includes: data subdomain and extension subdomain; The data subdomain is used to store data information; the extension subdomain is used to save the timestamp information corresponding to the processed data information.

6. The method according to claim 1, characterized in that The real-time streaming application configuration information includes: task number; Accordingly, based on the number of transactions per second and latency of the target operator stored in the distributed cache in different monitoring cycles, the transaction number per second performance curve and latency performance curve of the target operator are obtained, including: The total number of transactions per second of the target operator for all tasks of the same real-time stream job within each monitoring period is calculated as the total number of transactions per second of the target operator. Draw the target operator's transaction per second performance curve based on the total number of transactions per second corresponding to each monitoring period; The maximum latency of all tasks of the same target operator in the same real-time stream job within each monitoring period is taken as the total latency of the target operator. Draw a latency performance curve based on the total latency of the target operator corresponding to each monitoring period.

7. The method according to any one of claims 1 to 6, characterized in that While monitoring the cumulative data processing amount and cumulative processing time of the target operator in real time, it also includes: According to the real-time streaming application configuration information, the survival status and job restart count of at least one real-time streaming job are monitored in real time. When the real-time streaming job status is alive and the restart count is greater than 0, the restart count is reset to 0. When the real-time streaming job status is not alive and the restart count is greater than or equal to the set restart threshold, a prompt is issued indicating that the job cannot be automatically pulled up and restored; When the real-time streaming job status is not alive and the restart count is less than the set restart threshold, the current job is automatically submitted and the restart count is updated.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the real-time streaming application monitoring method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the monitoring method for real-time streaming applications according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the method for monitoring a real-time streaming application according to any one of claims 1 to 7.

Images