Visual management system and method for micro-server end-to-end link

By employing message queues, multi-dimensional correlation analysis, and real-time stream computing technologies in the microservice end-to-end link management system, combined with machine learning and WebGL visualization, the shortcomings of data collection and real-time analysis in microservice systems are addressed, achieving efficient data processing and intuitive visualization, and improving the system's monitoring capabilities and fault response speed.

CN118972231APending Publication Date: 2024-11-15GUIZHOU POWER GRID CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410863052.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing microservice end-to-end link management systems are inadequate in handling complex service call relationships and real-time responses, making it difficult to achieve efficient data collection, real-time analysis and processing, and intelligent alarms, resulting in insufficient system monitoring capabilities and fault response speed.

Method used

The system employs a message queue mechanism to asynchronously process microservice link data. It combines multi-dimensional correlation analysis algorithms, real-time stream computing technology, and machine learning algorithms to automatically capture microservice link data through a lightweight agent, generate pre-processed link data, and perform real-time analysis using network performance indicator analysis, traffic indicator statistics, and link alarm processing modules. Finally, it uses WebGL technology for multi-dimensional visualization.

Benefits of technology

It enables efficient collection and processing of large-scale microservice link data, rapid analysis of network performance and traffic metrics, dynamic optimization of alarm rules, and provides intuitive visualization and rich interactive operations for complex link data, thereby improving system monitoring capabilities and fault response speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972231A_ABST
    Figure CN118972231A_ABST
Patent Text Reader

Abstract

The invention discloses a visual management system and method for a micro-service end-to-end link, and relates to the technical field of visual management systems.The visual management system comprises a data collecting and processing module used for achieving asynchronous processing of data through a message queue in a micro-service architecture, and each micro-service conducts corresponding data processing according to the function of the micro-service; the data acquisition and processing module is connected with a network performance index analysis and processing module; the network performance index analyzing and processing module is used for processing network performance data, performing service association calculation, storing the data into a relational database and providing query and display functions; and the network performance index analysis processing module is connected with a flow index statistical analysis processing module. According to the invention, comprehensive monitoring and management of the micro-service end-to-end link are realized through cooperative work, a way for knowing the network performance condition in real time is provided for a user, and the link alarm is responded in time, so that the stability and reliability of the micro-service system are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of visual management systems, and in particular to a visual management system and method for an end-to-end link of a microservice. Background Art

[0002] Microservice end-to-end link refers to tracking and managing the entire request transmission path and calling process from the front-end to each microservice in the back-end in a complex microservice architecture. Its main function is to achieve comprehensive service monitoring, troubleshooting and performance optimization to ensure the stability and reliability of the microservice system. The usage environment is usually a large distributed system or microservice architecture, which contains multiple independently deployed microservice components that communicate and collaborate with each other through the network.

[0003] However, in the existing microservice end-to-end link architecture, there are many services and the calls between them are complex, so the end-to-end link is complicated. For systems with high real-time requirements, it is necessary to promptly detect and respond to anomalies or delays in the request path. Summary of the invention

[0004] In view of the deficiencies of the existing microservice end-to-end link management system in processing complex service call relationships and real-time responses, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to achieve efficient end-to-end link data collection, real-time analysis and processing, intelligent alarm and intuitive visualization in a complex microservice architecture, so as to improve the system's monitoring capability and fault response speed.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides a visual management system for an end-to-end link of a microservice, which includes a data acquisition and processing module, which uses a message queue mechanism to asynchronously process microservice link data, generates pre-processed link data, and transmits the pre-processed link data to a network performance indicator analysis and processing module; the network performance indicator analysis and processing module uses a multi-dimensional correlation analysis algorithm to receive pre-processed link data, extract network performance indicators and perform business correlation calculations, generate network performance analysis data, store the network performance analysis data in a distributed relational database, and transmit the network performance analysis data to a traffic indicator statistical analysis and processing module; the traffic indicator statistical analysis and processing module uses real-time stream computing technology to receive network performance analysis data, extract network traffic indicators and perform statistical analysis calculations to generate traffic statistical data, and transmit the traffic statistical data to a network performance indicator analysis and processing module. The data is stored in a distributed relational database, and the traffic statistics data is transmitted to the link alarm processing module; the link alarm processing module uses a machine learning algorithm to dynamically optimize the alarm rules, receive traffic statistics data, read and update the link alarm rules and indicators, calculate and analyze the traffic statistics data in real time according to the alarm rules, generate alarm information data, store the alarm information data in a high-performance cache database, and transmit the alarm information data to the data visualization module; the data visualization module uses interactive data visualization technology to read network performance analysis data and traffic statistics data from the distributed relational database, read alarm information data from the high-performance cache database, integrate network performance analysis data, traffic statistics data and alarm information data, and use WebGL-based 3D rendering technology to convert the integrated data into multi-dimensional visualization charts.

[0008] As a preferred solution for the visualization management system of the microservice end-to-end link described in the present invention, the workflow of the data acquisition and processing module is as follows: a lightweight agent is deployed on the microservice node to automatically capture microservice link data; the lightweight agent collects microservice link data in real time and preprocesses it; asynchronous I / O is used to send the preprocessed microservice link data in batches to the distributed message queue, and the microservice link data in the message queue is partitioned according to the microservice identifier; multiple consumer instances are started to process the microservice link data in the message queue in parallel; the consumer uses a sliding time window algorithm to aggregate and analyze the microservice link data, and converts the aggregated and analyzed microservice link data into a unified preprocessing format to generate preprocessed link data; the preprocessed link data is asynchronously transmitted to the network performance indicator analysis and processing module through the microservice interface; each link of the data acquisition and processing process is regularly checked, and the data acquisition frequency and batch size are adjusted based on the inspection results.

[0009] As a preferred solution for the visualization management system of the microservice end-to-end link described in the present invention, the workflow of the network performance indicator analysis and processing module is as follows: asynchronously receiving the pre-processed link data transmitted from the data acquisition and processing module through the microservice interface, and parsing and verifying the received pre-processed link data; extracting key network performance indicators from the pre-processed link data; applying a multi-dimensional correlation analysis algorithm to perform preliminary clustering and correlation analysis on the extracted key network performance indicators; combining predefined business rules, performing business correlation calculations, and identifying the mapping relationship between performance indicators and business processes; based on the correlation analysis results and business calculation results, generating network performance analysis data containing multi-dimensional information; asynchronously writing the generated network performance analysis data into a distributed relational database; asynchronously transmitting the network performance analysis data to the traffic indicator statistical analysis and processing module through the microservice interface; real-time monitoring of the processing performance of the network performance indicator analysis and processing module, and dynamically adjusting the parameters of the multi-dimensional correlation analysis algorithm.

[0010] As a preferred solution of the visualization management system of the microservice end-to-end link described in the present invention, the workflow of the traffic index statistical analysis processing module is as follows: receiving network performance analysis data in real time through the microservice interface, and using an adaptive sampling algorithm to dynamically adjust the receiving frequency of the network performance analysis data; applying a multi-scale feature extraction algorithm based on wavelet transform to extract multi-dimensional network traffic indicators from network performance analysis data; using the STL algorithm to decompose the multi-dimensional network traffic indicators into trend components, periodic components and random components; applying different statistical models to each component of the decomposed network traffic indicator, and dynamically adjusting the model parameters in combination with an online machine learning algorithm; performing multi-scale aggregation calculations, and generating traffic statistics data of different time granularities based on the decomposed and modeled network traffic indicators; writing the generated traffic statistics data into a distributed relational database in real time through an incremental update mechanism;

[0011] Through the priority queue mechanism, traffic statistics are asynchronously transmitted to the link alarm processing module; the flow calculation process is monitored in real time, and the feedback control algorithm is used to dynamically optimize the processing flow and resource allocation of network performance analysis data.

[0012] As a preferred solution for the visualization management system of the microservice end-to-end link described in the present invention, the workflow of the link alarm processing module is as follows: receiving traffic statistics data from the traffic indicator statistics analysis processing module in real time through the microservice interface; reading the current link alarm rules and indicators from the high-performance cache database; applying the reinforcement learning algorithm, based on the historical alarm data and the current traffic statistics data, dynamically optimizing the link alarm rules and indicators, and updating the optimized link alarm rules and indicators to the high-performance cache database; using the parallel computing framework, according to the optimized link alarm rules, the traffic statistics data are calculated and analyzed in real time; applying the anomaly detection algorithm based on the graph neural network to identify complex link anomaly patterns; using the adaptive threshold technology, generating alarm information data according to the detected anomalies; using the alarm aggregation algorithm, intelligently merging the alarm information data; storing the intelligently merged alarm information data in real time to the high-performance cache database, and asynchronously transmitting the alarm information data to the data visualization module through the microservice interface; applying the continuous learning mechanism, and continuously optimizing the alarm model and rules according to the operation and maintenance feedback.

[0013] As a preferred solution for the visualization management system of the end-to-end link of the microservice described in the present invention, the workflow of the data visualization module is as follows: reading network performance analysis data and traffic statistics data from a distributed relational database in parallel, and reading alarm information data from a high-performance cache database at the same time; using multi-threading technology to integrate network performance analysis data, traffic statistics data and alarm information data to form a high-dimensional data set; applying a principal component analysis algorithm to process the high-dimensional data set; using WebGL-based 3D rendering technology to convert the processed high-dimensional data set into a multi-dimensional visualization chart; using a real-time data push mechanism based on WebSocket to update the visualization chart; applying responsive layout technology to adjust the display of visualization charts on different devices, and developing a data drill-down function to support users to access the underlying data of the visualization chart.

[0014] As a preferred solution of the visualization management system of the microservice end-to-end link described in the present invention, the following steps are included in the dynamic optimization of link alarm rules and indicators: obtaining the initial link alarm rules and indicators, reading the historical alarm data and the current traffic statistics data from the high-performance cache database, and initializing the parameters of the reinforcement learning algorithm; defining the state s and the action a, the state s is the current network performance indicator combination, and the action a is the adjustment action for the link alarm rules and indicators; designing the reward function to evaluate the effect of the adjusted alarm rules and indicators; selecting the optimal or random action through the ε-greedy strategy, executing the selected action, and adjusting the alarm rules and indicators; monitoring the adjusted network performance indicators, observing the new state s', and calculating the instant reward r according to the new state s' and the historical alarm data; using the Q-learning algorithm to update the Q value, the specific formula is as follows:

[0015] Q(s,a)←Q(s,a)+β[r+γmaxQ(s',a')-Q(s,a)]

[0016] a'

[0017] Among them, Q(s,a) is the value of performing action a in state s, β is the learning rate, r is the immediate reward, γ is the discount factor, s' is the new state after performing the action, and a' is the action in the new state; based on the updated Q table or value function, the optimal action is selected to adjust the link alarm rules and indicators, and the optimized link alarm rules and indicators are updated to the high-performance cache database.

[0018] In the second aspect, an embodiment of the present invention provides a visual management method for end-to-end links of microservices, which includes a data acquisition and processing module that deploys a lightweight agent on a microservice node, captures and preprocesses microservice link data, generates preprocessed link data, and transmits it asynchronously through a message queue; a network performance indicator analysis and processing module receives and generates preprocessed link data, extracts network performance indicators, performs business association calculations, and generates network performance analysis data; a traffic indicator statistical analysis and processing module processes network performance analysis data in real time, extracts network traffic indicators, performs statistical analysis calculations, and generates traffic statistical data; a link alarm processing module receives traffic statistical data, reads and updates link alarm rules and indicators, performs real-time calculations and analysis on traffic statistical data, and generates alarm information data; integrates preprocessed link data, traffic statistical data, and alarm information data, and uses WebGL-based 3D rendering technology to convert the integrated data into multi-dimensional visual charts for display; and adjusts algorithm parameters and processing flow through real-time monitoring and feedback mechanisms.

[0019] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the visual management system of the microservice end-to-end link as described in the first aspect of the present invention are implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the visual management system of the microservice end-to-end link as described in the first aspect of the present invention are implemented.

[0021] The beneficial effects of the present invention are as follows: by adopting the message queue mechanism and parallel processing technology, the present invention realizes the efficient collection and processing of large-scale microservice link data. Combining multi-dimensional correlation analysis algorithms, real-time stream computing technology and machine learning algorithms, the present invention can quickly analyze network performance indicators and traffic indicators, and dynamically optimize alarm rules. By continuously monitoring the processing performance of each module and applying technologies such as adaptive sampling and dynamic parameter adjustment, the present invention can automatically adjust system parameters according to actual operating conditions. Utilizing WebGL-based 3D rendering technology and interactive data visualization technology, the present invention provides intuitive visualization of complex link data and rich interactive operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 Module connection diagram for the visual management system of the microservice end-to-end link.

[0024] Figure 2 Flowchart of the traffic indicator statistics analysis and processing module of the visual management system for the end-to-end link of microservices.

[0025] Figure 3 A computer device diagram for visually managing the end-to-end links of microservices. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0027] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0028] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0029] Example 1

[0030] Reference Figure 1 to Figure 3 , which is the first embodiment of the present invention, and provides a visual management system for an end-to-end link of a microservice, including:

[0031] like Figure 1 As shown in the figure, a module connection diagram of a visual management system for an end-to-end link of a microservice is given, including a data acquisition and processing module, a network performance indicator analysis and processing module, a traffic indicator statistical analysis and processing module, a link alarm processing module, and a data visualization module, wherein:

[0032] The data acquisition and processing module uses the message queue mechanism to asynchronously process the microservice link data, generate pre-processed link data, and transmit the pre-processed link data to the network performance indicator analysis and processing module.

[0033] Specifically, the workflow of the data acquisition and processing module is as follows: deploy a lightweight agent on the microservice node to automatically capture microservice link data; the lightweight agent collects microservice link data in real time and performs preprocessing, which includes data cleaning and formatting to ensure data consistency and accuracy; use asynchronous I / O to send the preprocessed microservice link data in batches to the distributed message queue, and partition the microservice link data in the message queue according to the microservice identifier; start multiple consumer instances to process the microservice link data in the message queue in parallel to ensure efficient data processing and load balancing; consumers use the sliding time window algorithm to aggregate and analyze the microservice link data, and convert the aggregated and analyzed microservice link data into a unified preprocessing format to generate preprocessed link data; through the microservice interface, the preprocessed link data is asynchronously transmitted to the network performance indicator analysis and processing module to ensure the efficiency and reliability of data transmission; regularly check each link of the data acquisition and processing process, monitor the performance indicators and bottlenecks of each link, and adjust the data acquisition frequency and batch size based on the inspection results to optimize data processing efficiency and resource utilization.

[0034] The network performance indicator analysis and processing module adopts a multi-dimensional correlation analysis algorithm, receives pre-processed link data, extracts network performance indicators and performs business correlation calculations, generates network performance analysis data, stores the network performance analysis data in a distributed relational database, and transmits the network performance analysis data to the traffic indicator statistical analysis and processing module.

[0035] Specifically, the workflow of the network performance indicator analysis and processing module is as follows: asynchronously receive the pre-processed link data transmitted from the data acquisition and processing module through the microservice interface, and parse and verify the received pre-processed link data to ensure the integrity and correctness of the data; extract key network performance indicators such as bandwidth, delay, packet loss rate, etc. from the pre-processed link data; apply the multi-dimensional correlation analysis algorithm to perform preliminary clustering and correlation analysis on the extracted key network performance indicators. The specific formula is as follows:

[0036]

[0037] Where S(i,j) is the correlation coefficient between network performance indicators i and j (used to measure the linear correlation between two indicators, where 1 indicates a complete positive correlation, -1 indicates a complete negative correlation, and 0 indicates no correlation), x ik 、x jk are the values ​​of network performance indicators i and j in the kth sample, respectively. are the average values ​​of network performance indicators i and j respectively, and n is the number of samples.

[0038] Furthermore, in combination with predefined business rules, business correlation calculations are performed to identify the mapping relationship between performance indicators and business processes; based on the correlation analysis results and business calculation results, network performance analysis data containing multi-dimensional information is generated; the generated network performance analysis data is asynchronously written to a distributed relational database; through a microservice interface, the network performance analysis data is asynchronously transmitted to a traffic indicator statistical analysis processing module; the processing performance of the network performance indicator analysis processing module is monitored in real time, and the parameters of the multi-dimensional correlation analysis algorithm are dynamically adjusted to adapt to network performance data of different scales and types.

[0039] The traffic indicator statistical analysis and processing module adopts real-time stream computing technology to receive network performance analysis data, extract network traffic indicators and perform statistical analysis and calculation, generate traffic statistical data, store the traffic statistical data in a distributed relational database, and transmit the traffic statistical data to the link alarm processing module.

[0040] Specifically, the workflow of the traffic indicator statistical analysis processing module is as follows: receive network performance analysis data in real time through the microservice interface, and use an adaptive sampling algorithm to dynamically adjust the frequency of receiving network performance analysis data. The specific formula is as follows:

[0041] f new =f old ×(1+δ)

[0042] Among them, f new is the new receiving frequency, f old is the current receiving frequency, and δ is the adjustment factor.

[0043] Furthermore, a multi-scale feature extraction algorithm based on wavelet transform is applied to extract multi-dimensional network traffic indicators from network performance analysis data. The wavelet transform formula is as follows:

[0044]

[0045] Among them, W ψ (a, b) are wavelet transform coefficients, x(h) is the signal, ψ is the wavelet function, c and b are the scale parameter and translation parameter respectively.

[0046] Furthermore, the STL algorithm is used to decompose the multi-dimensional network traffic indicators into trend components, periodic components and random components; different statistical models are applied to each component of the decomposed network traffic indicators, and the model parameters are dynamically adjusted in combination with the online machine learning algorithm; multi-scale aggregation calculations are performed to generate traffic statistics of different time granularities based on the decomposed and modeled network traffic indicators; the generated traffic statistics are written to the distributed relational database in real time through the incremental update mechanism; the traffic statistics are asynchronously transmitted to the link alarm processing module through the priority queue mechanism; the flow calculation process is monitored in real time, and the feedback control algorithm is used to dynamically optimize the processing flow and resource allocation of network performance analysis data.

[0047] The link alarm processing module uses machine learning algorithms to dynamically optimize alarm rules, receive traffic statistics, read and update link alarm rules and indicators, calculate and analyze traffic statistics in real time according to alarm rules, generate alarm information data, store the alarm information data in a high-performance cache database, and transmit the alarm information data to the data visualization module.

[0048] Specifically, the workflow of the link alarm processing module is as follows: receive traffic statistics data from the traffic indicator statistics analysis processing module in real time through the microservice interface; read the current link alarm rules and indicators from the high-performance cache database; apply the reinforcement learning algorithm to dynamically optimize the link alarm rules and indicators based on historical alarm data and current traffic statistics data, and update the optimized link alarm rules and indicators to the high-performance cache database; use the parallel computing framework to perform real-time calculation and analysis of traffic statistics data according to the optimized link alarm rules; apply the anomaly detection algorithm based on the graph neural network to identify complex link anomaly patterns; use the adaptive threshold technology to generate alarm information data based on the detected anomalies; use the alarm aggregation algorithm to intelligently merge the alarm information data to reduce alarm storms; store the intelligently merged alarm information data in real time to the high-performance cache database, and at the same time, asynchronously transmit the alarm information data to the data visualization module through the microservice interface; apply the continuous learning mechanism to continuously optimize the alarm model and rules based on operation and maintenance feedback.

[0049] It should be noted that the dynamic optimization of link alarm rules and indicators includes the following steps: obtaining the initial link alarm rules and indicators, reading historical alarm data and current traffic statistics from the high-performance cache database, and initializing the parameters of the reinforcement learning algorithm; defining state s and action a, where state s is the current combination of network performance indicators (such as packet loss rate, delay, and bandwidth utilization), and action a is the adjustment action for link alarm rules and indicators (for example, increasing or decreasing the packet loss rate threshold, delay threshold, and bandwidth utilization threshold). Design a reward function to evaluate the effect of the adjusted alarm rules and indicators (positive rewards will be obtained for identifying abnormal network behavior, and negative rewards will be obtained for false alarms or missed alarms); select the optimal or random action through the ε-greedy strategy, execute the selected action, and adjust the alarm rules and indicators; monitor the adjusted network performance indicators, observe the new state s', and calculate the immediate reward r based on the new state s' and historical alarm data; use the Q-learning algorithm to update the Q value, and the specific formula is as follows:

[0050] Q(s,a)←Q(s,a)+β[r+γmaxQ(s',a')-Q(s,a)]

[0051] a'

[0052] Among them, Q(s,a) is the value of performing action a in state s, β is the learning rate (controls the step size of each update), r is the immediate reward, γ is the discount factor (weighing the current reward and future rewards), s' is the new state after performing the action, and a' is the action in the new state.

[0053] Further, based on the updated Q table or value function, the optimal action is selected to adjust the link alarm rules and indicators, and the optimized link alarm rules and indicators are updated to the high-performance cache database.

[0054] It should be noted that the link alarm rule is a condition or threshold used to determine whether a network link is abnormal. For example, when certain performance indicators of a link exceed a preset threshold, an alarm will be triggered. These rules include but are not limited to: when the packet loss rate of the link exceeds a preset packet loss rate threshold (for example, 5%), an alarm is triggered; when the link delay (the time it takes for a data packet to travel from source to destination) exceeds a preset delay threshold, an alarm is triggered; when the link bandwidth utilization (the ratio of the actual bandwidth used to the total bandwidth) exceeds a preset bandwidth utilization threshold, an alarm is triggered. Link alarm indicators are specific performance indicators used to monitor link status, including but not limited to packet loss rate, delay, and bandwidth utilization.

[0055] The data visualization module uses interactive data visualization technology to read network performance analysis data and traffic statistics data from a distributed relational database, read alarm information data from a high-performance cache database, integrate network performance analysis data, traffic statistics data and alarm information data, and use WebGL-based 3D rendering technology to convert the integrated data into multi-dimensional visualization charts, and display interactive visualization results in real time on the user interface.

[0056] Specifically, the workflow of the data visualization module is as follows: read network performance analysis data and traffic statistics data from a distributed relational database in parallel, and read alarm information data from a high-performance cache database at the same time; use multi-threading technology to integrate network performance analysis data, traffic statistics data, and alarm information data to form a high-dimensional data set; apply the principal component analysis algorithm to process the high-dimensional data set; use WebGL-based 3D rendering technology to convert the processed high-dimensional data set into a multi-dimensional visualization chart; use a real-time data push mechanism based on WebSocket to update the visualization chart; build an interactive operation interface, which includes chart scaling, rotation, and data filtering functions; apply responsive layout technology to adjust the display of visualization charts on different devices, and develop data drill-down functions to support users to access the underlying data of visualization charts.

[0057] Furthermore, the present embodiment also provides a visual management method for end-to-end links of microservices, including a data acquisition and processing module that deploys a lightweight agent on a microservice node, captures and preprocesses microservice link data, generates preprocessed link data, and transmits it asynchronously through a message queue; a network performance indicator analysis and processing module receives and generates preprocessed link data, extracts network performance indicators, performs business association calculations, and generates network performance analysis data; a traffic indicator statistical analysis and processing module processes network performance analysis data in real time, extracts network traffic indicators, performs statistical analysis calculations, and generates traffic statistical data; a link alarm processing module receives traffic statistical data, reads and updates link alarm rules and indicators, performs real-time calculations and analysis on traffic statistical data, and generates alarm information data; integrates preprocessed link data, traffic statistical data, and alarm information data, and uses WebGL-based 3D rendering technology to convert the integrated data into multi-dimensional visualization charts for display; and adjusts algorithm parameters and processing procedures through real-time monitoring and feedback mechanisms.

[0058] This embodiment also provides a computer device, which is suitable for the visual management system of the end-to-end link of microservices, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the visual management system of the end-to-end link of microservices proposed in the above embodiment.

[0059] like Figure 3As shown, the computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the computer device housing, or an external keyboard, touch pad or mouse, etc.

[0060] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, a visualization management system for realizing an end-to-end link of microservices as proposed in the above embodiment is implemented.

[0061] In summary, by adopting the message queue mechanism and parallel processing technology, the present invention realizes the efficient collection and processing of large-scale microservice link data. Combining multi-dimensional correlation analysis algorithms, real-time stream computing technology and machine learning algorithms, the present invention can quickly analyze network performance indicators and traffic indicators, and dynamically optimize alarm rules. By continuously monitoring the processing performance of each module and applying technologies such as adaptive sampling and dynamic parameter adjustment, the present invention can automatically adjust system parameters according to actual operating conditions. Utilizing WebGL-based 3D rendering technology and interactive data visualization technology, the present invention provides intuitive visualization of complex link data and rich interactive operations. .

[0062] Example 2

[0063] Reference Figure 1 to Figure 3 , which is the second embodiment of the present invention, and this embodiment provides a visual management system for end-to-end links of microservices. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0064] To verify the actual effect of the present invention, a 14-day simulation experiment was conducted in the microservice architecture of a medium-sized e-commerce platform. The platform consists of 100 microservice nodes, and the average daily API call volume is about 10 million times. The experiment aims to compare the performance of the present system with that of the traditional monitoring system.

[0065] First, in terms of data acquisition and processing modules, this system shows significant advantages. By adopting the message queue mechanism and parallel processing technology, the system can process about 20,000 link data per second, which is 25% higher than the traditional system. Not only that, the average data processing delay has also been reduced from the original 180 milliseconds to 140 milliseconds. Especially during peak hours, by dynamically adjusting the sampling rate, the system can still maintain a data processing accuracy of more than 92%, which fully demonstrates its efficiency and stability.

[0066] Secondly, the improvement of the network performance indicator analysis and processing module and the traffic indicator statistical analysis and processing module has also brought significant results. In terms of anomaly detection, the accuracy of this system has reached 91%, which is significantly higher than the 84% of the traditional system. At the same time, the false alarm rate has been reduced from 10% to 6%, and the missed alarm rate has been reduced from 8% to 4.5%. More importantly, the average response time of the system has been shortened from 3 minutes to 90 seconds, greatly improving the speed of fault detection and processing.

[0067] The system also demonstrates superiority in the performance of the link alarm processing module. When processing complex call chains, it can accurately identify 88% of performance bottlenecks, while traditional systems can only identify 80%. In addition, in high-concurrency scenarios, system throughput increased by 20% and resource utilization increased by 15%, which fully demonstrates the efficiency and reliability of the system in processing large-scale data.

[0068] Finally, the improvement of the data visualization module also received positive feedback from users. Through the user experience survey, it was found that the average time for operation and maintenance personnel to use this system to diagnose problems was reduced from 25 minutes to 18 minutes. It is worth mentioning that 75% of users believe that the new visualization interface is more intuitive and can help them understand the complex dependencies between services more quickly. In addition, the overall availability of the system reached 99.95%, an increase of 0.05 percentage points from the previous level, further ensuring the continuity and reliability of monitoring.

[0069] In summary, the experimental data clearly show that the present invention has made substantial progress in improving the efficiency, accuracy and user experience of microservice architecture monitoring. Through the collaborative work of various modules, the system provides strong support for the operation and maintenance management of distributed architecture and effectively solves the monitoring problem in complex microservice environment.

[0070] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A visual management system for end-to-end links of microservices, characterized by: include, The data acquisition and processing module uses a message queue mechanism to asynchronously process microservice link data, generate pre-processed link data, and transmit the pre-processed link data to the network performance indicator analysis and processing module; A network performance indicator analysis and processing module adopts a multi-dimensional correlation analysis algorithm, receives the pre-processed link data, extracts network performance indicators and performs business correlation calculations, generates network performance analysis data, stores the network performance analysis data in a distributed relational database, and transmits the network performance analysis data to a traffic indicator statistical analysis and processing module; A traffic index statistical analysis processing module adopts real-time stream computing technology to receive the network performance analysis data, extract network traffic indexes and perform statistical analysis calculations to generate traffic statistical data, store the traffic statistical data in the distributed relational database, and transmit the traffic statistical data to the link alarm processing module; A link alarm processing module, which uses a machine learning algorithm to dynamically optimize alarm rules, receives the traffic statistics data, reads and updates the link alarm rules and indicators, calculates and analyzes the traffic statistics data in real time according to the alarm rules, generates alarm information data, stores the alarm information data in a high-performance cache database, and transmits the alarm information data to a data visualization module; The data visualization module adopts interactive data visualization technology, reads the network performance analysis data and the traffic statistics data from the distributed relational database, reads the alarm information data from the high-performance cache database, integrates the network performance analysis data, the traffic statistics data and the alarm information data, and uses WebGL-based 3D rendering technology to convert the integrated data into a multi-dimensional visualization chart.

2. The visual management system for microservice end-to-end links according to claim 1, characterized in that: The workflow of the data acquisition and processing module is as follows: Deploy lightweight agents on microservice nodes to automatically capture microservice link data; The lightweight agent collects microservice link data in real time and performs preprocessing; Use asynchronous I / O to send the pre-processed microservice link data in batches to the distributed message queue, and partition the microservice link data in the message queue according to the microservice identifier; Start multiple consumer instances to process microservice link data in the message queue in parallel; The consumer uses a sliding time window algorithm to aggregate and analyze the microservice link data, and converts the aggregated and analyzed microservice link data into a unified preprocessing format to generate preprocessed link data; Through the microservice interface, the pre-processed link data is asynchronously transmitted to the network performance indicator analysis and processing module; Regularly check each link of the data collection and processing process, and adjust the data collection frequency and batch size based on the inspection results.

3. The visual management system for microservice end-to-end links according to claim 1, characterized in that: The workflow of the network performance indicator analysis and processing module is as follows: Asynchronously receive the pre-processed link data transmitted from the data acquisition and processing module through the microservice interface, and parse and verify the received pre-processed link data; Extract key network performance indicators from pre-processed link data; Apply multi-dimensional correlation analysis algorithm to conduct preliminary clustering and correlation analysis on the extracted key network performance indicators; Combined with predefined business rules, business association calculations are performed to identify the mapping relationship between performance indicators and business processes; Generate network performance analysis data containing multi-dimensional information based on correlation analysis results and business calculation results; Asynchronously write the generated network performance analysis data into a distributed relational database; Through the microservice interface, the network performance analysis data is asynchronously transmitted to the traffic indicator statistical analysis processing module; Monitor the processing performance of the network performance indicator analysis and processing module in real time, and dynamically adjust the parameters of the multi-dimensional correlation analysis algorithm.

4. The visual management system for microservice end-to-end links according to claim 1, characterized in that: The workflow of the traffic index statistical analysis and processing module is as follows: Receive network performance analysis data in real time through the microservice interface, and use an adaptive sampling algorithm to dynamically adjust the frequency of receiving network performance analysis data; Apply a multi-scale feature extraction algorithm based on wavelet transform to extract multi-dimensional network traffic indicators from network performance analysis data; Use the STL algorithm to decompose the multi-dimensional network traffic indicators into trend components, periodic components and random components; Apply different statistical models to the components of the decomposed network traffic indicators, and dynamically adjust the model parameters in combination with online machine learning algorithms; Perform multi-scale aggregation calculations to generate traffic statistics at different time granularities based on decomposed and modeled network traffic indicators; The generated traffic statistics data are written into the distributed relational database in real time through the incremental update mechanism; Through the priority queue mechanism, traffic statistics are asynchronously transmitted to the link alarm processing module; Monitor the flow computing process in real time and use feedback control algorithms to dynamically optimize the processing flow and resource allocation of network performance analysis data.

5. The visual management system for microservice end-to-end links according to claim 1, characterized in that: The workflow of the link alarm processing module is as follows: Receive traffic statistics from the traffic indicator statistics analysis and processing module in real time through the microservice interface; Read the current link alarm rules and indicators from the high-performance cache database; Apply reinforcement learning algorithms to dynamically optimize link alarm rules and indicators based on historical alarm data and current traffic statistics, and update the optimized link alarm rules and indicators to a high-performance cache database; Use a parallel computing framework to perform real-time calculation and analysis of traffic statistics based on optimized link alarm rules; Apply anomaly detection algorithms based on graph neural networks to identify complex link anomaly patterns; Using adaptive threshold technology, alarm information data is generated based on detected abnormal conditions; Use alarm aggregation algorithm to intelligently merge alarm information data; The intelligently merged alarm information data is stored in real time in a high-performance cache database, and the alarm information data is asynchronously transmitted to the data visualization module through the microservice interface; Apply continuous learning mechanism to continuously optimize alarm models and rules based on operation and maintenance feedback.

6. The visual management system for end-to-end links of microservices according to claim 5, characterized in that: The workflow of the data visualization module is as follows: Read network performance analysis data and traffic statistics data from the distributed relational database in parallel, and read alarm information data from the high-performance cache database; Use multi-threading technology to integrate network performance analysis data, traffic statistics data, and alarm information data to form a high-dimensional data set; Apply principal component analysis algorithms to process high-dimensional data sets; Use WebGL-based 3D rendering technology to convert processed high-dimensional data sets into multi-dimensional visualization charts; Adopt real-time data push mechanism based on WebSocket to update visual charts; Apply responsive layout technology to adjust the display of visualization charts on different devices, and develop data drill-down functions to support users to access the underlying data of visualization charts.

7. The visual management system for microservice end-to-end links according to claim 5, characterized in that: The dynamic optimization of link alarm rules and indicators includes the following steps: Obtain initial link alarm rules and indicators, read historical alarm data and current traffic statistics from the high-performance cache database, and initialize the parameters of the reinforcement learning algorithm; Define state s and action a. State s is the current network performance indicator combination, and action a is the adjustment action for link alarm rules and indicators. Design reward functions to evaluate the effects of adjusted warning rules and indicators; Select the optimal or random action through the ε-greedy strategy, execute the selected action, and adjust the alarm rules and indicators; Monitor the adjusted network performance indicators, observe the new state s', and calculate the immediate reward r based on the new state s' and historical alarm data; Use the Q-learning algorithm to update the Q value. The specific formula is as follows: Q(s,a)←Q(s,a)+β[r+γmaxQ(s',a')-Q(s,a)] a' Where Q(s,a) is the value of executing action a in state s, β is the learning rate, r is the immediate reward, γ is the discount factor, s' is the new state after executing the action, and a' is the action in the new state; Based on the updated Q table or value function, the optimal action is selected to adjust the link alarm rules and indicators, and the optimized link alarm rules and indicators are updated to the high-performance cache database.

8. A visual management method for an end-to-end link of a microservice, based on the visual management system for an end-to-end link of a microservice according to any one of claims 1 to 7, characterized in that: Also includes, The data collection and processing module deploys a lightweight agent on the microservice node to capture and preprocess the microservice link data, generate preprocessed link data, and transmit it asynchronously through the message queue; The network performance indicator analysis and processing module receives and generates pre-processed link data, extracts network performance indicators, performs service association calculations, and generates network performance analysis data; The traffic index statistical analysis processing module processes network performance analysis data in real time, extracts network traffic indexes, performs statistical analysis calculations, and generates traffic statistics data; The link alarm processing module receives traffic statistics data, reads and updates link alarm rules and indicators, calculates and analyzes traffic statistics data in real time, and generates alarm information data; Integrate pre-processed link data, traffic statistics data, and alarm information data, and use WebGL-based 3D rendering technology to convert the integrated data into multi-dimensional visualization charts for display; Adjust algorithm parameters and processing procedures through real-time monitoring and feedback mechanism.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the visual management system of the microservice end-to-end link described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual management system of the microservice end-to-end link described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Visual monitoring system and method for broadband network

    CN120825416A

  • A Visual Monitoring System and Method for Broadband Networks

    CN120825416B

  • Streaming analysis method and system for satellite network stream detection data

    CN120934607A

  • Simulation data switching management system based on micro-service architecture

    CN122508595A