Method for carrying out data analysis through monitoring probe
Through the combination of OTel APIs and related components, the problem of diversity and integration difficulty of traditional monitoring solutions is solved, efficient data analysis and real-time health monitoring across systems are realized, and real-time data analysis and visual presentation capabilities are improved.
Patent Information
- Application Number
- CN202411306919.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-20
AI Technical Summary
Traditional monitoring and tracking solutions in cloud computing and distributed system environments are limited by the diversity of the technology stack, inconsistency of the underlying environment, inconsistency of data formats, and integration difficulty, resulting in reduced observability quality and limited performance. , resource consumption is too high, making it difficult to achieve efficient tracking and analysis across systems and languages.
Observation data is collected through OTel APIs components, data cleaning and filtering is performed, data aggregation and processing is performed using Exporter and Collector components, and real-time monitoring of service health is achieved with algorithm modules, and data chart rendering and displaying through VUE3 framework.
It realizes intelligent adjustment of sampling rate and real-time correlation analysis across data types, improves the real-time and visual presentation capabilities of data analysis, and provides detailed data support for platform operations.
Smart Images
Figure CN119166462A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data analysis, and in particular relates to a method for performing data analysis through a monitoring probe. Background Art
[0002] In recent years, microservice architecture has become the mainstream technical framework. In the process of microservice governance, data monitoring and data governance scenarios mostly use Istio technology to ensure that monitoring obtains service data.
[0003] Istio is an open source service mesh platform that can control how data is shared between microservices. By deploying a sidecar proxy in the environment, you can add Istio support to the service. Istio provides network establishment, load balancing, service-to-service authentication, monitoring and other functions. Use the Istio control plane function to configure and manage the proxy to intercept all network communications between microservices.
[0004] Although Istio can control microservices, it has high requirements for the K8S platform version. It requires the platform K8S version to be 1.18 or above to be used, and only supports weak control scenarios at the microservice architecture level. It cannot meet the strong control at the data model level. Istio needs to hijack service traffic, so there is a risk of affecting the normal function of the service.
[0005] When using Istio for monitoring, the performance indicators of the service cannot be obtained. To solve this problem, first check the configuration of Prometheus to ensure that Prometheus correctly collects the performance indicators of the service. You can view the collection rules in the Prometheus configuration file. Secondly, check the configuration of Grafana to ensure that Grafana correctly displays the performance indicators of the service. Finally, you can view the performance indicators in the Grafana dashboard.
[0006] In the current cloud computing and distributed system environment, as service architectures become increasingly complex, system observability has become a key factor in ensuring service quality, improving troubleshooting efficiency, and optimizing system performance. Traditional monitoring and tracing solutions are often limited by the diversity of technology stacks, the inconsistency of underlying environments, the inconsistency of data formats, and the difficulty of integration, resulting in reduced observability quality, limited performance, and excessive resource consumption, making it difficult to achieve efficient tracing and analysis across systems and languages. Summary of the invention
[0007] In view of the technical problems that the above-mentioned traditional monitoring and tracing solutions are often limited by the diversity of technology stacks, the inconsistency of underlying environments, the inconsistency of data formats and the difficulty of integration, the present invention provides a method for performing data analysis through a monitoring probe.
[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0009] A method for performing data analysis by monitoring probes comprises the following steps:
[0010] S1. With the help of the OTeI APIs component, embed OTeI API into the developed application, so that the application has the ability to collect observation data and set which data should be collected;
[0011] S2. Perform data cleaning operations on the data;
[0012] S3: The data generated by Exporter flows into Collector component for further filtering and aggregation.
[0013] S4: The data obtained from the backend service is then analyzed by the algorithm module.
[0014] The method of S2 for performing data cleaning operations on data is: using the OTeLDKs component closely related to the OTeL APIs to provide the ability to temporarily store and transmit data, breaking up, reorganizing, sampling, and filtering the data to clean the data, combining data from different data sources but with the same context to make the data fuller, and finally converting the data into a standard format with the help of OTLP and sending it to the outside.
[0015] The Collector component in S3 has batch processing capabilities. After the data is processed centrally, it is sent to the pre-prepared backend service with the help of the Exporter capability.
[0016] The algorithm module in S4 adopts a health algorithm model to calculate the health weight of the service, intuitively reflects the health of the system by establishing health indicators, models the health, and realizes a dynamic state of the health of microservices, solves the problems of cumbersome monitoring configuration and complex indicators of microservices, and realizes real-time monitoring of service health.
[0017] The method for calculating the health weight of a service is: calculating the health weight of the service based on the number of calls, accesses, and failures of the service.
[0018] The results of the data analysis are rendered into data charts by the VUE3 framework to complete the front-end display.
[0019] The Exporter is located in the OTeL Library, and separates the application and data backends, thereby enabling users to select and switch backends.
[0020] The Collector component is outside the entire OTeL Library process and is responsible for receiving and further filtering, aggregating, and batch processing data.
[0021] The OTeI SDKs component is the intermediate bridge between the API and Exporter, and completes the work of data filtering and event sampling by adding additional configurations.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] 1. The present invention intelligently adjusts the sampling rate according to factors such as the real-time load of the service, error rate, and newly emerged hot spots. For requests with frequent errors or newly launched functional modules, the sampling ratio is automatically increased to obtain more diagnostic information, while the sampling rate is reduced to save resources when the system is running smoothly.
[0024] 2. The present invention automatically associates data from different sources across multiple dimensions such as tracking ID, log timestamp, service tag, etc., to achieve real-time correlation analysis across data types (tracking, measurement, log).
[0025] 3. To address the data processing delay problem in large-scale distributed systems, the present invention can perform preliminary processing and filtering on the edge nodes where data is generated, and only transmit key or aggregated data to the centralized analysis platform, thereby improving the real-time performance of data analysis.
[0026] 4. The present invention classifies and marks the data received by OTeI Collector, and combines statistical algorithms and VUE3 framework rendering to achieve a comprehensive visualization of platform capability data on the front page. It provides convenient visualization of data in the daily operation of the platform and provides detailed data support for the decision-making of platform users. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0028] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0029] Figure 1 This is the OTeI data collection and analysis diagram of the present invention;
[0030] Figure 2 This is a flowchart of the working process of this embodiment. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0032] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0033] First, if Figure 1 As shown in the figure, with the help of the OTeI APIs component, the OTeI API is embedded into the developed application, so that the application has the ability to collect observation data and can also set which data should be collected. Given the language relevance of the OTeI APIs component, the specific operation steps may be different for different application languages, and this will also affect the degree of intrusion of this action on the application itself.
[0034] In the second step, the OTeI SDKs component, which is closely related to the OTeI APIs, can provide the ability to temporarily store and transmit data. In this process, developers can perform operations such as breaking up, reorganizing, sampling, filtering, etc. on the data to clean the data. They can also combine data from different data sources but with the same context to make the data more complete. Finally, with the help of OTLP, the data is converted into a standard format and sent to the outside.
[0035] In the third step, the data produced by the Exporter in the previous step can flow into the Collector component for further filtering, aggregation and other processing. The Collector generally also has batch processing capabilities. After the data is processed centrally, the Exporter's capabilities are used again to send the data to the pre-prepared backend service.
[0036] In the fourth step, the algorithm module calculates the health weight of the service based on the number of service calls, accesses, and failures obtained from the backend service to complete the data analysis. Finally, the VUE3 framework performs data chart rendering to complete the front-end display.
[0037] Furthermore, this embodiment adopts a health algorithm model to calculate the health weight of the service. By establishing health indicators, the health of the system can be intuitively reflected, and the health can be modeled to achieve a dynamic health state for microservices, solve the problems of cumbersome monitoring configuration and complex indicators of microservices, and realize real-time monitoring of service health.
[0038] like Figure 2 As shown in the figure, a cloud-native PaaS platform is deployed for an enterprise, and data information is obtained by using OTeI probes to achieve the function of data analysis.
[0039] Service A, service B, application C, etc. are registered on the PaaS platform. If you want to actively obtain analysis results such as the capacity scheduling status, daily capacity call times, and capacity health of A, B, and C, and the PaaS platform is for digital governments and industrial digital enterprises, it cannot affect the system performance while collecting data. Therefore, OTe technology is needed to achieve data monitoring and analysis.
[0040] First, with the help of the capabilities of the OTeI APIs component, the OTeI API is embedded into the developed PaaS platform, so that the application has the ability to collect observation data and is set to collect service A, service B, and application C data.
[0041] Secondly, after obtaining the data of service A, service B, and application C, developers can perform operations such as breaking up, reorganizing, sampling, and filtering the data to clean the data, and use OTLP to convert the data into a standard format and send it to the outside.
[0042] Then, the data produced by the Exporter in the previous step can flow into the Collector component for further filtering, aggregation and other processing. After the data is centrally processed, it can be sent to the pre-prepared backend service with the help of the Exporter's capabilities.
[0043] Finally, the service A, service B, and application C data obtained from the backend service are used to calculate the health weight of the service based on the algorithm model according to the number of service calls, accesses, and failures, and complete the data analysis. Finally, the VUE3 framework performs data chart rendering, and displays data analysis results such as capacity large concurrent scheduling analysis, capacity daily call number analysis, and capacity health analysis on the front end to provide data decision support for different users.
[0044] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.
Claims
1. A method for data analysis by monitoring probes, characterized in that: The following steps are involved: S1. With the help of the OTeI APIs component, embed OTeI API into the developed application, so that the application has the ability to collect observation data and set which data should be collected; S2. Perform data cleaning operations on the data; S3: The data generated by Exporter flows into Collector component for further filtering and aggregation. S4: The data obtained from the backend service is then analyzed by the algorithm module.
2. A method for data analysis by monitoring probe according to claim 1, characterized in that: The method of S2 for performing data cleaning operations on data is: using the OTeI SDKs component closely related to the OTeI APIs to provide the ability to temporarily store and transmit data, breaking up, reorganizing, sampling, and filtering the data to clean the data, combining data from different data sources but with the same context to make the data fuller, and finally converting the data into a standard format with the help of OTLP and sending it to the outside.
3. The method for performing data analysis by monitoring probe according to claim 1, characterized in that: The Collector component in S3 has batch processing capabilities. After the data is processed centrally, it is sent to the pre-prepared backend service with the help of the Exporter capability.
4. The method for performing data analysis by monitoring probe according to claim 1, characterized in that: The algorithm module in S4 adopts a health algorithm model to calculate the health weight of the service, intuitively reflects the health of the system by establishing health indicators, models the health, and realizes a dynamic state of the health of microservices, solves the problems of cumbersome monitoring configuration and complex indicators of microservices, and realizes real-time monitoring of service health.
5. The method for performing data analysis by monitoring probe according to claim 4, characterized in that: The method for calculating the health weight of a service is: calculating the health weight of the service based on the number of calls, accesses, and failures of the service.
6. The method for performing data analysis by monitoring probe according to claim 4, characterized in that: The results of the data analysis are rendered into data charts by the VUE3 framework to complete the front-end display.
7. The method for performing data analysis by monitoring probe according to claim 1, characterized in that: The Exporter is located in the OTeL Library, and separates the application and data backends, thereby enabling users to select and switch backends.
8. The method for performing data analysis by monitoring probe according to claim 1, characterized in that: The Collector component is outside the entire OTeL Library process and is responsible for receiving and further filtering, aggregating, and batch processing data.
9. The method for performing data analysis by monitoring probe according to claim 2, characterized in that: The OTeI SDKs component is the intermediate bridge between the API and Exporter, and completes the work of data filtering and event sampling by adding additional configurations.