Real-time monitoring method and system for operation and maintenance large screen

Abstract

The invention discloses an operation and maintenance large screen real-time monitoring method, which comprises the following steps: identifying a key path in a business process according to an actual business demand and scene, and carrying out dynamic point burying; the Agent carries out data acquisition and transmission; the Skywalking-OAP collects data to execute primary processing, and sends the data to a message queue system; and the measurement platform performs data processing after receiving, generates a corresponding response message, and transmits the response message back to the requester through a message queue. An operation and maintenance large screen monitoring system comprises a business data monitoring module, a business heartbeat monitoring module, a service interface monitoring module, a system state monitoring module, a database connection pool monitoring module, a third-party interaction monitoring module, an abnormal early warning data monitoring module and an operation and maintenance work monitoring module. According to the invention, the real-time and accurate acquisition of the system resource operation data is realized, and the expandability and flexibility of the system are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual operation and maintenance monitoring, and specifically relates to a real-time monitoring method and system for an operation and maintenance large screen. Background Art

[0002] As the scale and complexity of information technology systems continue to increase, more stringent requirements are placed on real-time monitoring of system operation status, fault warning, and performance optimization. Currently, mainstream operation and maintenance monitoring solutions generally use static data display methods and rely on manual data analysis and processing, which have the following limitations:

[0003] Lack of real-time performance. The display of system status is usually delayed and cannot immediately reflect the latest changes, which may lead to the neglect of potential problems.

[0004] The data processing capability is limited. When dealing with complex systems and multi-dimensional monitoring data, existing solutions are difficult to process data efficiently and present intuitive and actionable information.

[0005] The operation and maintenance costs are high. The deployment and maintenance of the system require a high investment of resources, especially when dealing with high concurrency and large-scale environments. Summary of the invention

[0006] The present invention aims to provide a real-time monitoring method and system for an operation and maintenance large screen, which is used to solve the problems in the prior art of insufficient real-time monitoring, incomplete dimensions, and inefficient and unstable data processing.

[0007] The present invention solves the above problems through the following technical solutions:

[0008] A real-time monitoring method for an operation and maintenance large screen includes:

[0009] Step S1: According to actual business needs and scenarios, identify the key paths in the business process (such as enterprise login, financing processing, review process, etc.), and dynamically track key operation points such as API calls, database access, and third-party service responses;

[0010] Step S2, Agent collects and transmits data:

[0011] Step S21, collecting relevant data according to pre-configured tracking rules;

[0012] Step S22, encapsulating the collected data into a specific format;

[0013] Step S23, using encryption and authentication mechanism to transmit data with SkyWalking-OAP;

[0014] Step S3, Skywalking-OAP collects data from different agents and immediately performs preliminary processing such as data aggregation, cleaning and conversion after receiving the data;

[0015] Step S4, Skywalking-OAP sends the processed data message to the message queue system;

[0016] Step S5: the data message arrives at the measurement platform, and the measurement platform performs a receiving operation, and then performs data analysis, executes business logic, calls related services, etc.;

[0017] Step S6: After the processing is completed, the measurement platform generates a corresponding response message and transmits the response message back to the requester or related service through the message queue.

[0018] Furthermore, in step S1, exception handling and error logs are also embedded to quickly locate the cause of the problem and perform repairs and optimizations in a timely manner.

[0019] Furthermore, in step S2, for data with high real-time requirements, the Agent uses real-time transmission to immediately transmit the data to the OAP; for data with large data volume or low real-time requirements, the Agent uses batch transmission to improve overall transmission efficiency.

[0020] Furthermore, in step S3, data processing and filtering rules may be customized according to specific business requirements to retain data that is critical to business analysis, thereby improving the efficiency and accuracy of data processing.

[0021] Furthermore, in step S4, the message queue can act as a data buffer to temporarily store data that cannot be processed immediately, effectively alleviating the processing pressure of Skywalking-OAP.

[0022] Furthermore, in step S4, the message queue adopts a load balancing mechanism to ensure that messages can be evenly distributed to multiple processing nodes, thereby avoiding the problem of single-point overload of processing nodes.

[0023] Furthermore, in step S5, for different types of events, the measurement service will execute specific processing logic, including data cleaning, event matching, format conversion, etc. After the processing is completed, the data will be stored in the redis and MySQL databases.

[0024] An operation and maintenance large-screen monitoring system, comprising:

[0025] Business data monitoring module, business heartbeat monitoring module, service interface monitoring module, system status monitoring module, database connection pool monitoring module, third-party interaction monitoring module, abnormal warning data monitoring module, and operation and maintenance work monitoring module.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0027] The present invention realizes real-time and accurate acquisition of system resource operation data through processing methods such as Skywalking-OAP and message queues; it adopts advanced data processing algorithms and optimization technologies, which can efficiently clean, convert and aggregate massive data to ensure the real-time and accuracy of the data; by combining with the real-time monitoring function of Skywalking-OAP and Prometheus, it can effectively track and analyze system performance to ensure the stable operation of the system; by separating data calculation from the query process, the system performance is further optimized, the operation and maintenance costs are reduced, and the scalability and flexibility of the system are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a flow chart of a method for real-time monitoring of a large-screen operation and maintenance system provided by the present invention;

[0029] Figure 2 A schematic diagram of an operation and maintenance large-screen monitoring system module provided by the present invention. DETAILED DESCRIPTION

[0030] Embodiment 1

[0031] A real-time monitoring method for an operation and maintenance large screen includes:

[0032] Step S1, according to actual business needs and scenarios, by using the API interface to dynamically track points, identify the key paths in the business process (such as enterprise login, financing processing, review process, etc.), and dynamically track key operation points such as API calls, database access and third-party service responses. At the same time, you can track exception handling and error logs to quickly locate the cause of the problem and make repairs and optimizations in a timely manner.

[0033] Step S2, Agent data collection and transmission:

[0034] Data collection: Collect relevant data according to pre-configured tracking rules to ensure the accuracy and completeness of the data;

[0035] Data encapsulation: effectively encapsulate the collected data into a specific format to facilitate subsequent transmission and analysis;

[0036] Secure transmission: Encryption and verification mechanisms are used for data transmission between Agent and SkyWalking-OAP, effectively preventing data from being intercepted or tampered with during transmission, ensuring data confidentiality and integrity;

[0037] Real-time transmission: For data with high real-time requirements, Agent uses real-time transmission to immediately transmit the data to OAP;

[0038] Batch transmission: For data with large volume or low real-time requirements, Agent can use batch transmission. By processing and transmitting data in batches, network transmission overhead is reduced and overall transmission efficiency is improved.

[0039] In step S3, Skywalking-OAP collects data from different agents and performs preliminary processing steps such as data aggregation, cleaning, and conversion immediately after receiving it to ensure the accuracy and availability of the data. It can customize data processing and filtering rules according to specific business needs, thereby retaining data that is critical to business analysis and improving the efficiency and accuracy of data processing.

[0040] Step S4, after completing data aggregation, analysis and other operations, SkyWalking-OAP sends these data messages to the message queue system. SkyWalking-OAP uses integrated message queue technology to decouple data collection and subsequent processing processes, ensuring that the data collection module can operate independently of the data processing module. The two can communicate effectively through message queues, giving the system higher flexibility and simplifying the expansion and maintenance process.

[0041] In addition, the buffering function and load balancing characteristics of the message queue can act as a data buffer in a high-concurrency environment, temporarily storing data that cannot be processed immediately, effectively alleviating the processing pressure of the OAP. At the same time, the load balancing mechanism of the message queue ensures that messages can be evenly distributed to multiple processing nodes, avoiding the single-point overload problem of the processing node, thereby improving the stability and reliability of the system.

[0042] Step S5: When the data message arrives at the measurement platform, the measurement platform performs a receiving operation to ensure the integrity and correctness of the message. Subsequently, the measurement platform processes the message, which involves steps such as parsing the message content, executing business logic, and calling related services.

[0043] Step S6: After the processing is completed, the measurement platform will generate corresponding response messages and transmit these response messages back to the requester or related services through the message queue.

[0044] For different types of events, the measurement service will execute specific processing logic, including data cleaning, event matching, format conversion, etc. After processing, the data will be stored in redis and MySQL databases; during the data processing process, events are divided into the following four types: counting events, cache events, time-consuming events, and process termination events.

[0045] Counting events mainly involve real-time tracking. Real-time tracking means that whenever a new event occurs in the system, the relevant records will be captured and counted immediately. This real-time monitoring mechanism ensures that every event can be recorded accurately, thereby ensuring the integrity and real-time nature of the data. For example, the business heartbeat module uses Spring Boot applications, Prometheus monitoring system, and scheduled tasks to monitor and record real-time tracking data.

[0046] The specific implementation steps are as follows: define a custom quantity indicator with the indicator type of gauge in Prometheus; configure the Prometheus endpoint so that Prometheus can access all exposed indicators through the endpoint; start the Spring Boot application and ensure that the endpoint is accessible through the specified port; create a scheduled task, which is responsible for extracting tracking data from the cache or other data sources and updating the previously defined Gauge indicator.

[0047] Cache events involve events that require real-time calculations. These events usually require calculating the difference between the current time and the time when a specific event occurred. In order to improve the efficiency of data processing, the time corresponding to these events is stored in the redis cache. When real-time calculations are required, the required time data can be directly obtained from the cache, thereby improving the overall processing speed.

[0048] Time-consuming events refer to those events whose duration can be calculated through the rule table. Specifically, the processing of such events involves calculating the difference between the occurrence times of two or more different events to obtain a specific time-consuming indicator.

[0049] The specific operation is to store the time-consuming calculation logic in the rule table. When a time-consuming indicator needs to be calculated, the system will query the occurrence time of the relevant event according to the definition in the rule table, and apply the corresponding calculation logic to obtain the time consumption, and store the calculated time consumption indicator in the database. Its advantage is that it realizes the separation of data query and calculation, making the system more flexible, easy to maintain and expand. Moreover, the time-consuming calculation is based on pre-defined rules, and the calculation process is automated, so it can not only improve the data processing efficiency, but also improve the accuracy of the time-consuming calculation.

[0050] Process termination events are a series of events at the final stage of a business process that mark the end of the process, and involve cold and hot data migration events. That is, at the end of a business process, by reasonably migrating and processing this data, data storage and use are effectively managed to ensure data integrity and availability.

[0051] Embodiment 2

[0052] An operation and maintenance large-screen monitoring system, comprising:

[0053] Business data monitoring module, business heartbeat monitoring module, service interface monitoring module, system status monitoring module, database connection pool monitoring module, third-party interaction monitoring module, abnormal warning data monitoring module, and operation and maintenance work monitoring module. Among them:

[0054] Business data monitoring module: real-time monitoring of key business indicators to ensure the accuracy and real-time nature of monitoring data. Through the capture and analysis of real-time data, the operation and maintenance team can promptly discover data anomalies and take prompt measures to ensure the smooth operation of the business. Business data trends are analyzed to provide data support for business decisions and help business growth and optimization. By comparing and analyzing historical data and real-time data, potential opportunities and risks for business development can be discovered, providing management with a scientific basis for decision-making.

[0055] Business heartbeat monitoring module: monitors the heartbeat signals of business components to ensure the normal operation of business services, and promptly detects and handles service suspension or abnormal stop situations. Heartbeat monitoring helps to evaluate the stability and reliability of business services and prevent potential service interruption risks. By real-time monitoring of the heartbeat signals of business components, the operation and maintenance team can quickly locate and handle service anomalies to ensure business continuity and stability.

[0056] Service interface monitoring module: monitors key indicators such as API interface response time, success rate, error rate, etc. Through continuous monitoring of API interface, the operation and maintenance team can promptly discover performance problems of the interface, take measures to optimize it, and improve the response speed of the overall system and user satisfaction.

[0057] System status monitoring module: System status monitoring involves two core areas, service downtime alarm and business interface error alarm. Service downtime alarm focuses on continuous monitoring of key services such as databases, Web servers and application servers to ensure that these services can run stably under various circumstances; while business interface error alarm focuses on errors that occur when the system business interface processes requests; when the system monitors downtime events, the interface error rate exceeds the set threshold or the interface performance is significantly reduced, the system will automatically activate the alarm mechanism and quickly notify the operation and maintenance team through various communication methods such as email, SMS, instant messaging, etc. The operation and maintenance team can promptly deal with system-level risks and failures by monitoring the system's operating status in real time to ensure the continuity of business processes.

[0058] Database connection pool monitoring module: monitors the usage of database connection pool, including the number of connections, idle connections, active connections, etc., to ensure sufficient database connection resources; analyzes connection pool performance bottlenecks, optimizes connection pool configuration, improves database access efficiency, and reduces resource consumption. Through real-time monitoring and analysis of the database connection pool, the operation and maintenance team can ensure sufficient and efficient use of database connection resources to avoid business interruptions due to connection problems.

[0059] Third-party interaction monitoring module: monitors the interaction with third-party services, including call success rate, response time, etc., to ensure that the third-party services that the business relies on are stable and reliable. Through real-time monitoring of third-party services, the operation and maintenance team can ensure the smooth progress of the business and avoid business interruptions caused by third-party service problems.

[0060] Abnormal warning data monitoring module: Through real-time monitoring of abnormal business warning data of each product line and each business process, problems can be discovered and solved promptly before they are formally reported, reducing the processing time of operation and maintenance work orders, providing good customer service and avoiding business losses.

[0061] Operation and maintenance work monitoring module: Operation and maintenance work order system monitoring and intelligent customer service session number monitoring. Operation and maintenance work order quantity monitoring counts and displays detailed information such as the number, type, and processing status of operation and maintenance work orders in real time, thereby providing an intuitive workload display for the operation and maintenance team. Through this real-time monitoring and analysis, the operation and maintenance team can clearly understand the current workload distribution and processing efficiency, thereby optimizing resource allocation and improving overall work efficiency; intelligent customer service session number monitoring focuses on the operation of the intelligent customer service system, and monitors and counts key indicators such as the total number of intelligent customer service sessions, the number of correct sessions, the number of incoming calls, and the number of incoming calls transferred to manual calls in real time. By analyzing these data, the operation and maintenance team can evaluate the accuracy of intelligent customer service, promptly discover and solve potential problems, ensure the efficient operation of the intelligent customer service system, and improve user experience.

Claims

1. A real-time monitoring method for an operation and maintenance large screen, characterized in that: include: Step S1: Identify the key paths in the business process based on actual business needs and scenarios, and dynamically track points through API calls, database access, and third-party service responses; Step S2, Agent performs data collection and transmission; Step S3, Skywalking-OAP collects data from different agents, and performs preliminary processing such as data aggregation, cleaning and conversion after receiving the data; Step S4, the Skywalking-OAP sends the processed data message to the message queue system; Step S5, the data message arrives at the measurement platform, and the measurement platform performs data analysis, executes business logic, calls related services, etc. after receiving the data message; Step S6: After the processing is completed, the measurement platform generates a corresponding response message and transmits the response message back to the requester through the message queue.

2. A method for real-time monitoring of an operation and maintenance large screen according to claim 1, characterized in that: The step S2 specifically includes: Step S21, collecting data according to the dynamic tracking rule; Step S22, encapsulating the data into a specific format; Step S23, encryption and authentication mechanism is adopted when data is transmitted between the SkyWalking-OAP.

3. A method for real-time monitoring of an operation and maintenance large screen according to claim 1, characterized in that: In step S1, exception handling and error logs are also dynamically embedded to quickly locate the cause of the problem.

4. A method for real-time monitoring of an operation and maintenance large screen according to claim 2, characterized in that: In the step S2, for data with high real-time requirements, the Agent adopts real-time transmission; for data with low real-time requirements and data with large data volume, the Agent adopts batch transmission.

5. The method for real-time monitoring of an operation and maintenance large screen according to claim 1 is characterized in that: In step S3, the Skywalking-OAP can customize data processing and filtering rules according to business needs and retain data important for business analysis.

6. A method for real-time monitoring of an operation and maintenance large screen according to claim 1, characterized in that: In step S4, the message queue can temporarily store data that cannot be processed immediately.

7. A method for real-time monitoring of an operation and maintenance large screen according to claim 6, characterized in that: In step S4, the message queue uses a load balancing mechanism to evenly distribute messages to multiple processing nodes.

8. The method for real-time monitoring of an operation and maintenance large screen according to claim 1 is characterized in that: In step S5, for different types of events, the measurement service processes the data including data cleaning, event matching, and format conversion. After the processing is completed, the data will be stored in a database.

9. An operation and maintenance large screen monitoring system, characterized in that: include: Business data monitoring module, business heartbeat monitoring module, service interface monitoring module, system status monitoring module, database connection pool monitoring module, third-party interaction monitoring module, abnormal warning data monitoring module, and operation and maintenance work monitoring module.

Images