Monitoring method, system and equipment based on Spark streaming computing framework and medium

Abstract

The invention discloses a monitoring method, system and device based on a Spark stream-oriented computing framework and a medium. The method comprises the steps that the name of a Spark stream processing program is configured; the ID of the Spark stream processing program is inquired; acquiring statistical data of the Spark Web UI, and acquiring an HTML (Hypertext Markup Language) file for displaying statistical information of the stream processing program; the method comprises the following steps of: extracting a Javascript statistical code snippet in an HTML (Hypertext Markup Language) file, and extracting a Javascript code snippet containing time sequence statistical data; the Javascript code snippets are executed, and data are extracted; monitoring data is assembled into a Prometheus format, and the extracted data is assembled into a Prometheus-compatible monitoring data format in combination with an ID, a name and a monitoring index class of a Spark stream processing program; and the monitoring data is stored in a Promeheus, and the assembled monitoring data is pushed to a Promeheus monitoring system. The performance indexes of the system can be fed back in real time through real-time monitoring, so that a worker can quickly recognize and solve the performance bottleneck and accurately position the problem, resource utilization is optimized, the processing efficiency is improved, and it is ensured that the Spark streaming computing framework can continuously maintain the optimal performance state.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing and monitoring, and relates to a monitoring method, system, equipment and medium based on a Spark streaming computing framework. Background Art

[0002] With the rapid development of big data technology, real-time data processing has become a core requirement for many enterprises and organizations. As a powerful big data processing framework, Spark's streaming computing modules (including Spark Streaming and Structured Streaming) have been widely used in the field of real-time data processing due to their high throughput and low latency. However, when using the Spark streaming computing framework for real-time data analysis and processing, how to effectively monitor the running status of the stream processing program to ensure stable, efficient and real-time processing of data has become a technical problem that needs to be solved urgently. There are currently a variety of monitoring methods for the Spark streaming computing framework, but each method has certain limitations and is difficult to meet the requirements of comprehensiveness, real-timeness and accuracy of monitoring in actual production environments.

[0003] Spark's built-in Web UI provides a graphical interface for viewing the running status and performance indicators of stream processing tasks. However, this method has obvious flaws. First, the monitoring data in the Web UI is retained for a short time, and once the stream processing program is restarted, the previous monitoring data will be lost, and the ability to analyze historical data cannot be provided. Secondly, the monitoring data generated by the WebUI is embedded in the HTML page in the form of JavaScript code and cannot be directly obtained through the Spark API interface. This limits the integration and automated processing capabilities of the monitoring data, and it is difficult to connect to the existing monitoring system for unified management.

[0004] Another monitoring method is to capture various events in the stream processing process by implementing the Streaming Query Listener in the Spark streaming computing framework, and then achieve monitoring. Although this can obtain detailed stream processing event information, the implementation is complex and requires writing a special plug-in and installing it in the Spark engine. Since event information usually appears in pairs, it is necessary to handle the sequence and abnormal situations of event information, which increases the difficulty of implementation. At the same time, the introduction of plug-ins may affect the execution performance of Spark. If the plug-in needs to be modified, the overall configuration of Spark needs to be changed, which increases the maintenance cost.

[0005] Spark also provides a Metrics System for exposing monitoring indicator data. Although the Metrics System can expose monitoring data in a format compatible with monitoring systems such as Prometheus, the indicator name contains the ID of the stream processing program. Since the ID of the stream processing program will change after restart, the indicator name will also change. It is difficult to effectively connect the data before and after the restart in the monitoring system, which limits the ability of data analysis and trend prediction.

[0006] In summary, the existing technologies have problems such as short data retention time, high integration difficulty, complex implementation, performance impact, and poor data continuity when monitoring the Spark streaming computing framework. Therefore, developing a more efficient, comprehensive, and easy-to-use monitoring mechanism to solve the monitoring problem of the Spark streaming computing framework in real-time data processing has important practical significance and application value. Summary of the invention

[0007] The purpose of the present invention is to solve the problems of short data retention time, great integration difficulty, complex implementation, performance impact and poor data continuity when monitoring the Spark streaming computing framework in the prior art, and to provide a monitoring method, system, device and medium based on the Spark streaming computing framework.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: A monitoring method based on the Spark streaming computing framework includes the following steps: Configure the name of the Spark stream processing program and set the name of the stream processing program to be monitored through the Spark parameter; Query the ID of the Spark stream processing program. Use the API provided by Hadoop Yarn to query the list of currently running applications and find the ID of the corresponding Spark stream processing program through the configured stream processing program name. If the ID cannot be obtained, mark the program as exited. If the ID can be obtained, mark it as normal and continue to perform subsequent monitoring steps. Get the statistics of the Spark Web UI. According to the ID of the Spark stream processing program or the preset Web UI path, access the corresponding Web UI page and download or read the HTML file that displays the statistics of the stream processing program. Extract the Javascript statistical code snippet in the HTML file, read the HTML file content line by line, extract and save the Javascript code snippet containing time series statistical data; Execute the Javascript code snippet and extract the data. Execute the extracted Javascript code snippet to parse and extract the data of the indicator of interest. Assemble monitoring data into Prometheus format. Combine the extracted data with the ID, name, and monitoring indicator class of the Spark stream processing program to assemble it into a Prometheus-compatible monitoring data format. Save monitoring data to Prometheus and push assembled monitoring data to the Prometheus monitoring system for real-time monitoring, alarming, and data analysis.

[0009] The query Spark stream processing program ID is specifically: Obtain the address of Yarn ResourceManager and determine the network address and port number of Yarn ResourceManager through the Yarn Web UI page or the configuration items in the cluster configuration file; Use the Yarn REST API to query the application list. Construct and send an HTTP GET request to the ResourceManager REST API endpoint to obtain a list of all current applications in the Yarn cluster. Parse the response data, receive the response of the HTTP GET request, which contains detailed information of multiple applications, and extract the ID, name, type and status of each application; Find the Spark stream processing program, and traverse the multiple obtained applications to find the application records that match the specified Spark stream processing program name; Get the application ID and determine the status. If a matching application is found, extract its ID and determine its status. If the status indicates that the program is running, mark the program status as normal. If the status indicates that the program has exited or an error has occurred, mark the program status as exited or an error accordingly. If no matching application record is found, mark the program status as exited, indicating that the program has exited.

[0010] The statistical data of the Spark Web UI is obtained as follows: Construct an access URL. Use the access path of the Spark Web UI and the ID of the Spark stream processing program to construct the URL of the Web UI page for accessing statistical information. Access the Web UI page and get the HTML file. Use an HTTP client to send a request to the URL. Receive an HTTP response containing an HTML page showing Spark Streaming statistics. Download or read an HTML file showing stream handler statistics.

[0011] The Javascript statistical code snippet extracted from the HTML file is specifically: Read the HTML file content line by line; Start saving flag detection. For each line of content read, check whether it contains <script>标签，如果检测到<script>标签，则将接下来的行视为Javascript代码的开始，并准备开始保存内容；保存Javascript代码片段：从检测到<script>标签的下一行开始，将每一行的内容保存到一个字符串或缓冲区中，直到遇到< / script> Tags till now; End saving and processing. When a tag is encountered, the saving operation is stopped and the saved string or buffer content is regarded as a complete Javascript code snippet; Extract time series statistics. In a saved Javascript snippet, use string search, regular expression matching, or prefix matching to find the variable declaration lines that contain time series statistics.

[0012] The assembly monitoring data is in Prometheus format, specifically: Data extraction, extracting monitoring data from the Spark streaming program's Web UI and other monitoring methods, including the streaming application's survival status, data input rate, data processing rate, batch processing time, and data processing latency; Data tagging: adding the ID and name of the current Spark streaming program as a label to the extracted monitoring data to distinguish the monitoring data of different programs or instances; Assemble the Prometheus format and assemble the marked monitoring data into a Prometheus compatible format according to the Prometheus data model.

[0013] A monitoring system based on the Spark streaming computing framework includes the following modules: Configuration module, used to configure the name of the Spark stream processing program to be monitored, and set the name through Spark parameters; A query module, which uses the API interface provided by Hadoop Yarn to query the list of applications in the current running state, and searches for the ID of the corresponding Spark stream processing program in the list according to the configured stream processing program name; if the ID cannot be obtained, the program is marked as exited; if the ID can be obtained, it is marked as normal and the subsequent monitoring steps are continued; The data acquisition module accesses the corresponding Spark Web UI page according to the ID of the Spark stream processing program or the preset Web UI path, and downloads or reads the HTML file that displays the statistics of the stream processing program; A data extraction module, which parses the HTML file, reads the file content line by line, and extracts Javascript code snippets containing time series statistical data; A data processing module, wherein the data processing module executes the extracted Javascript code snippet, parses and extracts the monitoring indicator data of interest; The data assembly module combines the extracted monitoring data with the ID, name and monitoring indicator class of the Spark stream processing program, assembles them according to the data format requirements of Prometheus, and generates a monitoring data format compatible with Prometheus; The data storage and push module pushes the assembled monitoring data to the Prometheus monitoring system for storage.

[0014] The system also includes a monitoring strategy management module, which allows users to define and manage different monitoring strategies, including monitoring frequency, data collection method, and alarm triggering conditions, and is customized for different types of Spark stream processing programs or different monitoring requirements.

[0015] The system also includes a monitoring data visualization customization module, which allows users to customize the visualization interface of monitoring data, including selecting monitoring indicators to be displayed, adjusting chart styles, and setting layouts.

[0016] A device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method as described in any one of the preceding items when executing the computer program.

[0017] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the preceding items.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The monitoring method based on the Spark streaming computing framework in the present invention can instantly feedback the performance indicators of the system, such as processing speed, throughput, latency, etc., through real-time monitoring, so that system administrators and developers can quickly identify and solve performance bottlenecks. By accurately locating the problem, the configuration parameters of the Spark job can be dynamically adjusted to optimize resource utilization and improve processing efficiency, ensuring that the Spark streaming computing framework can continue to maintain the best performance state.

[0019] The monitoring method has the ability to respond quickly, can promptly detect and report system failures or abnormal conditions, and quickly notify relevant personnel through the alarm mechanism, which helps to reduce system interruption time and improve system stability and reliability. At the same time, monitoring data can also help conduct root cause analysis to find out the root cause of the failure, providing strong support for future prevention and maintenance. In some cases, the monitoring system can also integrate an automated recovery mechanism to automatically perform recovery operations, further reducing the impact of failures on the system.

[0020] Monitoring data trend analysis: Monitoring tools such as Prometheus can save monitoring data for a long time and show the changing trends of relevant monitoring data such as real-time input data, scheduling delay, processing time, and total delay. It helps developers understand system behavior, predict future performance, and make corresponding adjustments accordingly. Monitoring data trends also provide strong data support for system upgrades, expansions, or optimizations, making decisions more scientific and reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of the monitoring method based on the Spark streaming computing framework of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1 , is a flow chart of the monitoring method based on the Spark streaming computing framework in the present invention, which specifically includes the following steps: S1, configure the name of the Spark stream processing program, and set the name of the stream processing program to be monitored through the Spark parameter. The name is used to describe the function or purpose of the program, so as to facilitate identification and distinction in the subsequent monitoring process.

[0027] S2, query the ID of the Spark stream processing program, use the API interface provided by Hadoop Yarn to query the list of applications in the current running state, and find the ID of the corresponding Spark stream processing program through the configured stream processing program name; if the ID cannot be obtained, mark the program as exited; if it can be obtained, mark it as normal and continue to execute subsequent monitoring steps.

[0028] S2.1, obtain the address of Yarn ResourceManager, and determine the network address and port number of Yarn ResourceManager through the configuration items on the Yarn Web UI page or in the cluster configuration file; S2.2, use the Yarn REST API to query the application list, construct and send an HTTP GET request to the ResourceManager REST API endpoint to obtain a list of all current applications in the Yarn cluster; S2.3, parse the response data, receive the response of the HTTP GET request, which contains detailed information of multiple applications, and extract the ID, name, type and status of each application; S2.4, searching for a Spark stream processing program, and searching for an application record matching the specified Spark stream processing program name among the obtained multiple applications; S2.5, obtain the application ID and determine the status. If a matching application is found, extract its ID and determine its status. If the status indicates that the program is running, mark the program status as normal. If the status indicates that the program has exited or an error has occurred, mark the program status as exited or an error accordingly. If no matching application record is found, mark the program status as exited, indicating that the program has exited.

[0029] S3, obtain the statistical data of the Spark Web UI, access the corresponding Web UI page according to the ID of the Spark stream processing program or the preset Web UI path, and download or read the HTML file that displays the statistical information of the stream processing program.

[0030] S3.1, construct an access URL, using the access path of the Spark Web UI and the ID of the Spark stream processing program to construct a URL for accessing the Web UI page for statistical information; S3.2, access the Web UI page and obtain the HTML file, using the HTTP client to send a request to the URL; S3.3, receiving an HTTP response, the response including HTML page content showing statistics of the Spark streaming program; S3.4, download or read an HTML file showing statistics of the stream processing program.

[0031] S4, extracts the Javascript statistical code snippet in the HTML file, reads the HTML file content line by line, extracts and saves the Javascript code snippet containing time series statistical data.

[0032] S4.1, read the HTML file content line by line; S4.2, start saving flag detection, for each line of content read, check whether it contains <script>标签，如果检测到<script>标签，则将接下来的行视为Javascript代码的开始，并准备开始保存内容；S4.3，保存Javascript代码片段：从检测到<script>标签的下一行开始，将每一行的内容保存到一个字符串或缓冲区中，直到遇到< / script> tag; during the saving process, it does not include <script>标签行和< / script> The label row itself.

[0033] End saving and processing. When a tag is encountered, the saving operation is stopped and the saved string or buffer content is regarded as a complete Javascript code snippet; Extract time series statistics. In a saved Javascript snippet, use string search, regular expression matching, or prefix matching to find the variable declaration lines that contain time series statistics.

[0034] For example, var v1=[{"x": 1722994980000, "y": 0.0},{"x": 1722995040000, "y":0.0}}] (x: timestamp, y: value), one variable per line, so you can use string regular expressions or prefix matching to extract the line containing "var v1" for calculation.

[0035] S5, executes the Javascript code snippet and extracts data, executes the extracted Javascript code snippet, parses and extracts data of the indicators of interest, such as data input rate, data processing rate, batch processing time, data processing delay, etc.

[0036] S6, assembles monitoring data into Prometheus format, combines the extracted data with the ID, name and monitoring indicator class of the Spark stream processing program, and assembles it into a Prometheus compatible monitoring data format.

[0037] S6.1, data extraction, extracts monitoring data from the Web UI of the Spark streaming program and through other monitoring methods, including the survival status of the streaming application, data input rate, data processing rate, batch processing time, and data processing delay; S6.2, data tagging, adding the ID and name of the current Spark stream processing program as a label to the extracted monitoring data to distinguish the monitoring data of different programs or instances; S6.3, assemble the Prometheus format, according to the Prometheus data model, assemble the labeled monitoring data into a Prometheus compatible format. The Prometheus format usually includes metric name, label, timestamp and value.

[0038] S7 saves the monitoring data to Prometheus and pushes the assembled monitoring data to the Prometheus monitoring system for real-time monitoring, alarming, and data analysis.

[0039] An embodiment of the present invention is a monitoring system based on the Spark streaming computing framework, comprising the following modules: The configuration module is used to configure the name of the Spark stream processing program to be monitored. The name is set through the Spark parameter.

[0040] The query module uses the API interface provided by Hadoop Yarn to query the application list of the current running status, and searches for the ID of the corresponding Spark stream processing program in the list through the configured stream processing program name; if the ID cannot be obtained, the program is marked as exited; if the ID can be obtained, it is marked as normal and the subsequent monitoring steps are continued.

[0041] The data acquisition module accesses the corresponding Spark Web UI page according to the ID of the Spark stream processing program or the preset Web UI path, and downloads or reads the HTML file that displays the statistical information of the stream processing program.

[0042] A data extraction module parses the HTML file, reads the file content line by line, and extracts Javascript code snippets containing time series statistical data.

[0043] A data processing module executes the extracted Javascript code snippet, parses and extracts the monitoring indicator data of interest.

[0044] The data assembly module combines the extracted monitoring data with the ID, name and monitoring indicator class of the Spark stream processing program, assembles them according to the data format requirements of Prometheus, and generates a monitoring data format compatible with Prometheus.

[0045] The data storage and push module pushes the assembled monitoring data to the Prometheus monitoring system for storage.

[0046] The monitoring strategy management module allows users to define and manage different monitoring strategies, including monitoring frequency, data collection method, and alarm trigger conditions, and customize them for different types of Spark stream processing programs or different monitoring requirements.

[0047] The monitoring data visualization customization module allows users to customize the visualization interface of monitoring data, including selecting monitoring indicators to be displayed, adjusting chart styles, and setting layouts.

[0048] An embodiment of the present invention provides a terminal device. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0049] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0050] The device / terminal equipment may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The device / terminal equipment may include, but is not limited to, a processor and a memory.

[0051] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0052] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the apparatus / terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0053] If the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0054] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A monitoring method based on Spark streaming computing framework, characterized in that: The following steps are involved: Configure the name of the Spark stream processing program and set the name of the stream processing program to be monitored through the Spark parameter; Query the ID of the Spark stream processing program. Use the API provided by Hadoop Yarn to query the list of currently running applications and find the ID of the corresponding Spark stream processing program through the configured stream processing program name. If the ID cannot be obtained, mark the program as exited. If the ID can be obtained, mark it as normal and continue to perform subsequent monitoring steps. Get the statistics of the Spark Web UI. According to the ID of the Spark stream processing program or the preset Web UI path, access the corresponding Web UI page and download or read the HTML file that displays the statistics of the stream processing program. Extract the Javascript statistical code snippet in the HTML file, read the HTML file content line by line, extract and save the Javascript code snippet containing time series statistical data; Execute the Javascript code snippet and extract the data. Execute the extracted Javascript code snippet to parse and extract the data of the indicator of interest. Assemble monitoring data into Prometheus format. Combine the extracted data with the ID, name, and monitoring indicator class of the Spark stream processing program to assemble it into a Prometheus-compatible monitoring data format. Save monitoring data to Prometheus and push assembled monitoring data to the Prometheus monitoring system for real-time monitoring, alarming, and data analysis.

2. A monitoring method based on the Spark streaming computing framework as claimed in claim 1, characterized in that: The query Spark stream processing program ID is specifically: Obtain the address of Yarn ResourceManager and determine the network address and port number of Yarn ResourceManager through the Yarn Web UI page or the configuration items in the cluster configuration file; Use the Yarn REST API to query the application list. Construct and send an HTTP GET request to the ResourceManager REST API endpoint to obtain a list of all current applications in the Yarn cluster. Parse the response data, receive the response of the HTTP GET request, which contains detailed information of multiple applications, and extract the ID, name, type and status of each application; Find the Spark stream processing program, and traverse the multiple obtained applications to find the application records that match the specified Spark stream processing program name; Get the application ID and determine the status. If a matching application is found, extract its ID and determine its status. If the status indicates that the program is running, mark the program status as normal; If the status indicates that the program has exited or an error has occurred, the program status is marked as exited or an error; If no matching application record is found, the program status is marked as exited, indicating that the program has exited.

3. A monitoring method based on the Spark streaming computing framework as claimed in claim 1, characterized in that: The statistical data of the Spark Web UI is obtained as follows: Construct an access URL. Use the access path of the Spark Web UI and the ID of the Spark stream processing program to construct the URL of the Web UI page for accessing statistical information. Access the Web UI page and get the HTML file. Use an HTTP client to send a request to the URL. Receive an HTTP response containing an HTML page showing Spark Streaming statistics. Download or read an HTML file showing streamer statistics.

4. A monitoring method based on the Spark streaming computing framework as claimed in claim 1, characterized in that: The Javascript statistical code snippet extracted from the HTML file is specifically: Read the HTML file content line by line; Start saving flag detection. For each line of content read, check whether it contains <script>标签，如果检测到<script>标签，则将接下来的行视为Javascript代码的开始，并准备开始保存内容；保存Javascript代码片段：从检测到<script>标签的下一行开始，将每一行的内容保存到一个字符串或缓冲区中，直到遇到< / script> Tags till now; End saving and processing. When a tag is encountered, the saving operation is stopped and the saved string or buffer content is regarded as a complete Javascript code snippet; Extract time series statistics. In a saved Javascript snippet, use string search, regular expression matching, or prefix matching to find the variable declaration lines that contain time series statistics.

5. The monitoring method based on the Spark streaming computing framework as claimed in claim 1, characterized in that: The assembly monitoring data is in Prometheus format, specifically: Data extraction, extracting monitoring data from the Spark streaming program's Web UI and other monitoring methods, including the streaming application's survival status, data input rate, data processing rate, batch processing time, and data processing latency; Data tagging: adding the ID and name of the current Spark streaming program as a label to the extracted monitoring data to distinguish the monitoring data of different programs or instances; Assemble the Prometheus format and assemble the marked monitoring data into a Prometheus compatible format according to the Prometheus data model.

6. A monitoring system based on the Spark streaming computing framework, characterized in that: Includes the following modules: Configuration module, used to configure the name of the Spark stream processing program to be monitored, and set the name through Spark parameters; A query module, which uses the API interface provided by Hadoop Yarn to query the list of applications in the current running state, and searches for the ID of the corresponding Spark stream processing program in the list according to the configured stream processing program name; if the ID cannot be obtained, the program is marked as exited; if the ID can be obtained, it is marked as normal and the subsequent monitoring steps are continued; The data acquisition module accesses the corresponding Spark Web UI page according to the ID of the Spark stream processing program or the preset Web UI path, and downloads or reads the HTML file that displays the statistical information of the stream processing program; A data extraction module, which parses the HTML file, reads the file content line by line, and extracts Javascript code snippets containing time series statistical data; A data processing module, wherein the data processing module executes the extracted Javascript code snippet, parses and extracts the monitoring indicator data of interest; The data assembly module combines the extracted monitoring data with the ID, name and monitoring indicator class of the Spark stream processing program, assembles them according to the data format requirements of Prometheus, and generates a monitoring data format compatible with Prometheus; The data storage and push module pushes the assembled monitoring data to the Prometheus monitoring system for storage.

7. A monitoring system based on the Spark streaming computing framework as described in claim 6, characterized in that: The system also includes a monitoring strategy management module, which allows users to define and manage different monitoring strategies, including monitoring frequency, data collection method, and alarm triggering conditions, and is customized for different types of Spark stream processing programs or different monitoring requirements.

8. A monitoring system based on the Spark streaming computing framework as claimed in claim 6, characterized in that: The system also includes a monitoring data visualization customization module, which allows users to customize the visualization interface of monitoring data, including selecting monitoring indicators to be displayed, adjusting chart styles, and setting layouts.

9. A device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images