A multi-dimensional analysis system based on massive request data

Abstract

This invention discloses a multi-dimensional analysis system based on massive request data, belonging to the field of data analysis technology. The system includes: a quality display module, a full-link log module, a Zabbix module, a cloud monitoring module, and a monitoring module. This invention is mainly aimed at various scenarios in the field of massive data monitoring, and is primarily applicable to multi-dimensional analysis and monitoring systems that integrate traffic monitoring and performance monitoring of mixed data sources. This invention performs multi-dimensional analysis on massive request data from mixed data sources, effectively extracts the collected data, forms mutual correlations of information, establishes a mapping relationship between platform monitoring and upper-layer business, realizes the visualization of large-scale data display and alarm data from multiple data sources, promotes targeted optimization of various systems, optimizes monitoring coverage, and improves monitoring from a single level to multiple dimensions, further ensuring the stability of the entire request lifecycle and improving service stability.

Description

Technical Field

[0001] This invention belongs to the field of data analysis technology, and specifically relates to a multi-dimensional analysis system based on massive request data. Background Technology

[0002] In environments such as platform marketing, trending news, online education platforms, social media platforms, WeChat and Douyin mini-programs, transportation platforms, and public health and safety, scenarios with sudden surges in QPS are commonplace. When UV (unique visitors) rapidly rises to the millions, it poses a severe challenge to system operation and fault diagnosis. Traditional monitoring methods simply collect and indiscriminately record data. The system scales up to accommodate the surge in traffic based on the expected QPS peak, but it fails to extract regular and effective information from the monitoring data. Furthermore, monitoring methods cannot intuitively reflect the system's physical and business topology, requiring deeper analysis of error information: involving troubleshooting, analysis, and dependency analysis.

[0003] Prior art 1: CN201410606547.6 discloses a method and system for multi-dimensional statistical analysis of massive DNS logs. The system includes a DNS data acquisition module, a DNS data aggregation center, a cloud server, a DNS log processing center, a DNS log statistical analysis center, and a database. The DNS log statistical analysis center described in this patent is used to statistically analyze the processed DNS log data based on multiple given dimensions, generating multi-dimensional DNS log data statistical analysis results, and storing the obtained results in the database. The purpose of this patent is to solve the problem of decreased data processing efficiency (see patent specification 0052); however, the disclosed system does not further explain how to generate alerts or how to plan the next steps based on the processing results.

[0004] Prior art 2: CN201611052099.5 discloses a multi-dimensional processing method for massive data, which can obtain the latest business data analysis results within minutes and reflect business changes in real time. Its data processing module is only used for data aggregation (see patent specification 0014); the disclosed system does not further explain how to issue alerts or how to make further plans based on the processing results.

[0005] Prior art 3: CN201510004956.3 provides a website threat detection system based on real-time logs. This system includes a real-time log data acquisition unit, a real-time log data conversion and distribution unit, a real-time log data processing and detection unit, and a real-time log data monitoring unit. The purpose of this invention is to improve the processing capability and timeliness of log data, enhance the timeliness of website threat detection and response, reduce the risks of website operation, and provide strong protection for network information security (please refer to the specification of this invention patent 0046).

[0006] Prior art 4: CN201510812126.3 discloses a method, corresponding system, and data processing method for integrating a distributed data processing system; it discloses an integrated distributed processing system, including: a Hadoop cluster, a distributed database HBase configured in the cluster, a full-text search server Solr, and a distributed real-time computing system Storm, and a unified data access interface configured on the data access layer of the cluster; wherein, the distributed database HBase, the full-text search server Solr, and the distributed real-time computing system Storm perform data processing through a first configuration file and a second configuration file configured on the data access layer of the cluster (see specification 0051); the purpose of this invention is to achieve advantages such as diversified processing and low integration cost, and its essential purpose is to improve efficiency.

[0007] Based on the above four existing technologies, it can be found that the problems with the existing technologies are concentrated in:

[0008] 1. Simply collect and record data without differentiation. The system can adapt to the surge in traffic by scaling up based on the expected peak QPS. This is because existing research focuses on how to improve the processing efficiency of massive request data.

[0009] 2. No regular and effective information was extracted from the monitoring data, and the monitoring methods could not intuitively reflect the topology of the system's physical and business layers. A deeper analysis of error information is needed, involving troubleshooting, analysis, and dependency analysis. Existing technologies often utilize high-efficiency distributed clusters to handle massive amounts of information and ensure the timeliness of the final results, but they lack further research and analysis on how to deal with the results. For example, when a website encounters anomalies, threats, or attacks, it can react promptly, but this is merely feedback and cannot be used to automatically counterattack based on a mechanism. Summary of the Invention

[0010] (I) Technical Solution

[0011] This invention is achieved through the following technical solution: a multi-dimensional analysis system based on massive request data, the system comprising:

[0012] Quality display module: used for data input and display;

[0013] End-to-end logging module: used for collecting and parsing logs;

[0014] Zabbix module: Used to monitor hardware process status;

[0015] Cloud monitoring module: used for monitoring sites, services, hosts, logs, and custom functions;

[0016] The system also includes:

[0017] Monitoring module: used for multi-source data acquisition, business indicator monitoring, anomaly analysis and handling;

[0018] The monitoring module includes:

[0019] Alarm submodule: Used for data alarms; includes: monitor alarm unit, zabbix alarm unit, and comprehensive alarm unit;

[0020] Processing submodule: Used to send alarm information according to alarm level, automatically process alarm events, and visualize alarm processing; includes follow-up automation unit, analysis automation unit, and observability unit;

[0021] The analysis automation unit includes a fault simulation subunit and a fault learning subunit.

[0022] As a further explanation of the above solution, the quality display module is used to display the core indicators of the access layer and the core indicators of the service layer.

[0023] As a further explanation of the above solution, the end-to-end log module collects logs from the access layer and business layer through Filebeat, parses the logs through Logstash, aggregates them into ELK, and finally displays and queries them through the end-to-end log built on Kibana.

[0024] As a further explanation of the above solution, the cloud monitoring module monitors metrics including connection count, memory usage, CPU usage, master-slave synchronization, and slow query count.

[0025] As a further explanation of the above scheme, the monitor alarm unit is specifically as follows:

[0026] Alarm threshold selection strategy based on sample data;

[0027] Set at least one threshold according to the alarm threshold policy;

[0028] An alarm will be triggered after the threshold is set.

[0029] The alarm threshold strategy includes business indicator strategy, sample comparison strategy, and trend prediction strategy;

[0030] The thresholds include cumulative value, average value, maximum value, and minimum value.

[0031] As a further explanation of the above scheme, the Zabbix alarm unit includes a system-level alarm subunit and a service-level alarm subunit;

[0032] The alarm indicators of the system-level alarm subunit include: CPU utilization, load, disk utilization, and memory utilization.

[0033] The alarm indicators of the service layer alarm sub-unit include: connection count, service process, and port liveness alarms.

[0034] As a further explanation of the above scheme, the integrated alarm unit includes: Sentry alarm subunit, dial test alarm subunit, modulated alarm subunit, and personalized alarm subunit;

[0035] The dial-up alarm subunit is used to test whether the access layer link is unobstructed by triggering and replaying historical requests of the access layer through a timed task.

[0036] The modulated alarm subunit is used for quality monitoring and alarming of the logic layer and storage layer.

[0037] The personalized alarm subunit is used for personalized alarms that are strongly coupled based on specific domain knowledge.

[0038] As a further explanation of the above scheme, the alarm level specifically sends alarm information as follows:

[0039] Activate the alarm based on the set alarm level or non-response time;

[0040] Alarm notifications are sent according to preset times;

[0041] The alarm notification history will be saved and used to generate reporting information.

[0042] As a further explanation of the above solution, the automatic processing of alarm events and the visualization of alarm processing are specifically as follows:

[0043] Received an alarm event;

[0044] The system automatically analyzes and classifies the data;

[0045] Based on the execution of the alarm event handling strategy;

[0046] The classification includes alarm events that can be resolved automatically and alarm events that require user action.

[0047] As a further explanation of the above solution, the alarm events that can be automatically resolved are specifically as follows:

[0048] The specific alarm events that can be automatically resolved are as follows:

[0049] The fault simulation subunit imports preset fault information and preset fault handling solution data;

[0050] The fault learning subunit establishes a fault learning model;

[0051] Receive actual fault information and import it into the fault learning subunit to analyze and obtain fault handling results;

[0052] The results are then visualized.

[0053] (III) Beneficial Effects

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1. This invention provides low-cost tracking points, constructs a user-friendly browsing and query experience, ensures high reliability of data reading and writing, and improves the ease of use and effectiveness of centralized logging systems, centralized metrics systems, and distributed tracing systems.

[0056] 2. By analyzing alarm event information, the system automatically analyzes and makes preliminary judgments, classifying alarm events into those that can be resolved automatically and those that require user action. This further reduces manpower and improves system stability.

[0057] 3. By introducing fault simulation and fault learning, the fault simulation disrupts the system in a predetermined manner, and the entire process from the occurrence of the fault to its handling and resolution is recorded in the fault learning system to form a pattern record for storage. When an actual fault occurs, alarm events can be transmitted to the fault learning system for identification. The fault learning system can simulate fault resolution or display the fault resolution process for reference by the operations and maintenance or development teams. Attached Figure Description

[0058] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0059] Figure 1 This is a schematic diagram of the system framework of Embodiment 1 of the present invention. Detailed Implementation

[0060] Example

[0061] A multi-dimensional analysis system based on massive request data, the system comprising:

[0062] The Quality Display Module is used for data access and display. It displays core metrics of both the access layer and the business layer. It's important to further explain that the Quality Display Module plays a crucial role in load testing, monitoring, and early problem detection. The QPS (Queries Per Second) from the traffic entry point to the access layer is a key QPS metric to monitor. The access layer includes core metrics such as connection wait time and status codes. Real-time data display at the business layer allows for querying based on business criteria, observing business growth, comparing historical 7-day sample values, and focusing on core business metrics such as inventory and order status.

[0063] The end-to-end logging module is used to collect and parse logs. It collects logs from the access layer and business layer via Filebeat, parses them using Logstash, aggregates them in ELK, and finally displays and queries them using the end-to-end log system built on Kibana. It should be further noted that, in addition to access layer logs, considering the potential for slow Elasticsearch queries due to excessive log volume caused by a surge in business layer traffic, a certain proportion of logs are sampled and stored in Elasticsearch for querying by the end-to-end logging system.

[0064] The Zabbix module is used to monitor hardware process status. It should be further explained that the purpose of this module is to automatically discover and register devices through Zabbix network discovery. With the help of automatic discovery rule templates and filters, it can accurately realize the automatic discovery of business service processes and automatically add monitoring items and triggers.

[0065] The cloud monitoring module is used for monitoring sites, services, hosts, logs, and custom metrics. These metrics include connection count, memory usage, CPU usage, master-slave synchronization, and slow query count. It's worth noting that commonly used metrics include connection count, memory usage, CPU usage, master-slave synchronization, and slow query count. Combined with data layer monitoring via the quality dashboard, this provides a comprehensive understanding of the data layer's operational status.

[0066] The system also includes:

[0067] The monitoring module is used for multi-source data acquisition, business indicator monitoring, anomaly analysis, and handling. It's worth noting that this monitoring system, integrating real-time business monitoring, intelligent alarms, and fault analysis, utilizes a multi-source acquisition mechanism (monitor) to periodically collect and store monitoring data in InfluxDB. The monitoring panel can be configured via monitor to display historical 7-day sample values ​​for comparison, and these values ​​are synchronized to the quality display. Based on a comparison strategy using sample data and a trend prediction strategy, when an access layer business anomaly occurs, the abnormal access log can be queried using the status code to obtain the trace_id. The trace_id can then be used to query the abnormal link for anomaly analysis. Zabbix is ​​used to monitor host CPU, memory, hard disk, IO, connection count, and service process status.

[0068] The monitoring module includes:

[0069] Alarm submodule: Used for data alarms; includes: monitor alarm unit, zabbix alarm unit, and comprehensive alarm unit;

[0070] The monitor alarm unit operates as follows:

[0071] Alarm threshold selection strategy based on sample data;

[0072] Set at least one threshold according to the alarm threshold policy;

[0073] An alarm will be triggered after the threshold is set.

[0074] The alarm threshold strategy includes business indicator strategy, sample comparison strategy, and trend prediction strategy;

[0075] The thresholds include cumulative value, average value, maximum value, and minimum value.

[0076] The monitor can be configured from different perspectives, such as cumulative, average, maximum, or minimum values, and will issue an alarm after a threshold is triggered. The monitor settings can be based on alarm threshold strategies based on sample data comparison or predicted trend comparison, and different dimensions of alarms can be set depending on the data collection source.

[0077] The Zabbix alarm unit includes a system-level alarm subunit and a service-level alarm subunit.

[0078] The alarm indicators of the system-level alarm subunit include: CPU utilization, load, disk utilization, and memory utilization.

[0079] The high-value table of the service layer alarm subunit includes: connection count, business process, and port liveness alarms.

[0080] Zabbix alerts are divided into system-level and service-level alerts. System-level alerts include basic metrics such as CPU utilization, load, disk utilization, and memory utilization, all of which are system-level host monitoring alerts. Service-level alerts include connection count, process and port liveness alerts. Zabbix automatic discovery rules are used, and templates are bound to automatically discover process and port information, creating monitoring items and triggers. Comprehensive and efficient monitoring and alerting are provided for situations such as OutOfMemoryError (OOM), zombie processes, and process deadlocks.

[0081] The integrated alarm unit includes: Sentry alarm subunit, dial-up alarm subunit, modulated alarm subunit, and personalized alarm subunit;

[0082] The dial-up test alarm subunit is used to verify the access layer link's connectivity by triggering and replaying historical requests at a scheduled time. The dial-up test alarm verifies the access layer link's connectivity by triggering and replaying historical requests at a scheduled time. Due to uncertainties such as environment and network conditions, there may be parsing anomalies, network congestion, request timeouts, etc. Therefore, a dial-up test mechanism is added to ensure the smooth operation of the entire link.

[0083] The modulated alarm subunit is used for quality monitoring alarms in the logic layer and storage layer. Modulated alarms are mainly used for quality monitoring alarms of backend services such as logic layer and storage layer. The dial-up test alarm and Zabbix service liveness detection only focus on network connectivity and not on service success rate. The reliability of the modulated alarm is improved by filtering dial-up test and liveness detection requests.

[0084] The personalized alarm subunit is used for personalized alarms that are strongly coupled with specific domain knowledge. Examples of personalized alarms that are strongly coupled with specific domain knowledge include order inventory alarms and automatic kill alarms for slow query timeouts in the data layer (SQL).

[0085] Processing submodule: Used to send alarm information according to alarm level; automatically process alarm events and visualize alarm processing; includes follow-up automation unit, analysis automation unit, and observability unit;

[0086] The analysis automation unit includes a fault simulation subunit and a fault learning subunit.

[0087] It should be further explained that the alarm level specifically sends alarm information as follows:

[0088] Activate the alarm based on the set alarm level or non-response time;

[0089] Alarm notifications are sent according to preset times;

[0090] The alarm notification history will be saved and used to generate reporting information.

[0091] Based on the alarm level or prolonged lack of response, activate the Level 1 alarm, sending SMS and phone notifications every 5-10 minutes. Send a daily alarm event report and a weekly alarm event summary. Optimize personnel scheduling to improve efficiency and accuracy.

[0092] The steps for automatically handling alarm events and visualizing alarm handling are as follows:

[0093] Received an alarm event;

[0094] The system automatically analyzes and classifies the data;

[0095] Based on the execution of the alarm event handling strategy;

[0096] The system automatically analyzes and initially categorizes alarm events into those that can be resolved automatically and those that require user intervention. The system then selects whether an alarm event requires automatic resolution or user action.

[0097] The steps for automatically resolving alarm events are as follows:

[0098] The fault simulation subunit imports preset fault information and preset fault handling solution data;

[0099] The fault learning subunit establishes a fault learning model;

[0100] Receive actual fault information and import it into the fault learning subunit to analyze and obtain fault handling results;

[0101] The results are then visualized.

[0102] The specific fault simulation subunit disrupts the system in a predetermined manner, and the entire process from fault occurrence to fault handling and resolution is recorded in the fault learning subunit to form a pattern record for storage. When an actual fault occurs, alarm events can be transmitted to the fault learning subunit for identification. The fault learning subunit can simulate fault resolution or display the fault resolution process for reference by the operations and maintenance or development teams.

[0103] The standard description of this module is as follows:

[0104] When(Event)

[0105] The IF(Condition1) fault learning subunit identifies and classifies faults.

[0106] THEN(Action1) enables automated resolution of alarm events;

[0107] ELSE(Action2)OR(NULL) indicates that an alarm event requiring user action is being passed to the user for further processing.

[0108] The advantages of this invention are:

[0109] 1. This invention provides low-cost tracking points, constructs a user-friendly browsing and query experience, ensures high reliability of data reading and writing, and improves the ease of use and effectiveness of centralized logging systems, centralized metrics systems, and distributed tracing systems.

[0110] 2. By analyzing alarm event information, the system automatically analyzes and makes preliminary judgments, classifying alarm events into those that can be resolved automatically and those that require user action. This further reduces manpower and improves system stability.

[0111] 3. By introducing fault simulation and fault learning, the fault simulation disrupts the system in a predetermined manner, and the entire process from the occurrence of the fault to its handling and resolution is recorded in the fault learning system to form a pattern record for storage. When an actual fault occurs, alarm events can be transmitted to the fault learning system for identification. The fault learning system can simulate fault resolution or display the fault resolution process for reference by the operations and maintenance or development teams.

[0112] The control method of this invention is to control the device by manually starting and stopping the switch. The wiring diagram of the power element and the supply of power are common knowledge in the field. Since this invention is mainly used to protect mechanical devices, the control method and wiring layout will not be explained in detail.

[0113] The control method of this invention is automatic control through a controller. The control circuit of the controller can be implemented by simple programming by those skilled in the art. The power supply is also common knowledge in the art. Furthermore, since this invention is mainly used to protect mechanical devices, the control method and circuit connection will not be explained in detail here.

[0114] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0115] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-dimensional analysis system based on mass request data, characterized in that, The system comprises: A quality display module, which is configured to display core indicators of an access layer and core indicators of a service layer; A full-link log module, which is configured to collect and analyze logs; the full-link log module collects logs from the access layer and the service layer through filebeat, analyzes the logs through logstash, and aggregates the logs to ELK, and finally displays and queries the logs based on Kibana; in addition to access layer access logs, a certain proportion of the logs are sampled and saved to ES, and provided to the full-link log system for querying; A Zabbix module, which is configured to monitor hardware process states; the Zabbix module automatically discovers and registers devices through network discovery, and cooperates with automatic discovery rule templates and filters to automatically discover business service processes, and automatically adds monitoring items and triggers; A cloud monitoring module, which is configured to monitor sites, services, hosts, logs and custom monitoring; the cloud monitoring module monitors indicators including the number of connections, memory usage, CPU usage, master-slave synchronization, and the number of slow queries; The system further comprises: A monitoring module, which is configured to collect data from multiple sources, monitor business indicators, analyze and handle exceptions; the monitoring module collects data from multiple sources through monitor, stores the monitoring data in InfluxDB at regular intervals, configures a monitoring panel through monitor, and displays a comparison of historical 7-day sample values, and synchronizes the keys to the quality display; the monitoring module compares sample data and predicts trends based on a comparison strategy, and when an access layer business exception occurs, the monitoring module can query the exception access log through the status code and obtain the trace_id, query the exception link through the trace_id, and analyze the exception; the monitoring module comprises: An alarm submodule, which is configured to alarm data; the alarm submodule comprises a monitor alarm unit, a zabbix alarm unit, and a comprehensive alarm unit; A processing submodule, which is configured to send alarm information according to alarm levels, automatically handle alarm events, and visualize alarm processing; the processing submodule comprises a follow-up automation unit, an analysis automation unit, and an observability unit; The analysis automation unit comprises a fault simulation subunit and a fault learning subunit; The monitor alarm unit specifically comprises the following: selecting an alarm threshold strategy based on sample data; Setting at least one threshold value according to the alarm threshold strategy; and alarming when the threshold value is triggered; The alarm threshold strategy comprises a business indicator strategy, a sample comparison strategy, and a prediction trend strategy; the threshold value comprises a cumulative value, an average value, a maximum value, and a minimum value; The zabbix alarm unit comprises a system layer alarm submodule and a service layer alarm submodule; The system layer alarm subunit alarm index includes: CPU usage, load, disk usage, memory usage; The high net worth table of the service layer alarm subunit includes: connection number, service process and port survival alarm; Zabbix alarm is divided into system layer and service layer; The system layer includes system host level monitoring alarm; The alarm of the service layer includes connection number, service process and port survival alarm; Automatic discovery rules are used to bind templates to automatically discover business processes and ports, and monitoring items and triggers are created; The comprehensive alarm unit includes: a Sentry alarm subunit, a dial test alarm subunit, a model adjustment alarm subunit and a personalized alarm subunit; The dial test alarm subunit is used to trigger historical requests of the access layer through a timing task and a timing playback, to verify whether the access layer link is smooth; The dial test alarm triggers historical requests of the access layer through a timing task and a timing playback, to verify whether the access layer link is smooth; The model adjustment alarm subunit is used for quality monitoring alarm of the logic layer and the storage layer; The model adjustment alarm is used for quality monitoring alarm of the background service, and focuses on business success rate, to improve the credibility of the model adjustment alarm by filtering dial test and survival detection requests; The personalized alarm subunit is used for personalized alarm coupled with specific domain knowledge; The personalized alarm is coupled with personalized alarm according to specific domain knowledge. 2.The multi-dimensional analysis system based on massive request data according to claim 1, wherein the alarm level sends alarm information as follows: according to the set alarm level or the unresponsive time, starting an alarm; sending an alarm notification according to a preset time; and saving alarm notification history to form report information. 3.The multi-dimensional analysis system based on massive request data according to claim 1, wherein the automatic processing of alarm events and alarm processing visualization are as follows: receiving an alarm event; automatically analyzing and classifying by the system; and executing an alarm event processing strategy. 4.The multi-dimensional analysis system based on massive request data according to claim 3, wherein the automatically solvable alarm event is as follows: a fault simulation subunit imports preset fault information and preset fault processing solution data; a fault learning subunit establishes a fault learning model; actual fault information is received and imported into the fault learning subunit to analyze and obtain a fault processing result; and the result is visualized. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images