Equipment performance data real-time statistics and intelligent analysis system and equipment

Abstract

The invention provides an equipment performance data real-time statistics and intelligent analysis system and equipment, in the system, a data acquisition layer is used for acquiring and preprocessing equipment performance data of target equipment, and a real-time storage and calculation layer is used for performing streaming processing on the preprocessed equipment performance data to obtain real-time statistical indexes and storing the real-time statistical indexes; and the AI intelligent analysis layer is used for carrying out anomaly identification on target equipment through a dynamically updated long short-term memory (LSTM) network model based on the real-time statistical index to obtain anomaly identification result data, and carrying out root positioning to obtain root positioning result data if an abnormal index exists. According to the method, millisecond-level real-time equipment abnormal performance analysis from data acquisition to result output can be realized, the AI model with dynamic learning ability is associated with multi-dimensional data, the identification accuracy and root positioning efficiency of implicit anomalies can be remarkably improved, meanwhile, the system architecture is simplified, and the deployment and maintenance cost can be effectively reduced.

Description

Technical Field

[0001] This application relates to the field of equipment monitoring and maintenance technology, and in particular to a system and equipment for real-time statistical and intelligent analysis of equipment performance data. Background Technology

[0002] With the acceleration of digital transformation, the scale of complex systems such as internet applications, industrial control systems, and financial trading platforms continues to expand, and the daily performance data volume generated by a single system has jumped from the GB level to the TB level. Real-time monitoring of system performance status and timely detection of performance bottlenecks (such as CPU overload and memory leaks) are core requirements for ensuring stable system operation and improving user experience. For example, if a financial trading system experiences a response latency exceeding 100ms due to performance anomalies, it may lead to transaction failures; if an industrial control system cannot identify equipment performance degradation in real time, it may cause production downtime.

[0003] Current performance data processing technologies in the industry have significant shortcomings: on the one hand, traditional data collection intervals are mostly 5-30 seconds, which cannot capture instantaneous performance fluctuations; on the other hand, analysis methods rely on manually preset thresholds (such as alarms when CPU utilization exceeds 85%), making it difficult to identify hidden problems (such as the risk of leakage due to slowly increasing memory usage), and problem localization requires manual correlation of multi-dimensional data, which is extremely inefficient.

[0004] Therefore, developing a performance data processing system that integrates AI (Artificial Intelligence) technology and possesses both real-time and intelligent analysis capabilities has become a key direction for solving industry pain points. Summary of the Invention

[0005] In view of this, embodiments of this application provide a system and device for real-time statistical and intelligent analysis of device performance data, in order to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of this application provides a real-time statistical and intelligent analysis system for device performance data, comprising: a data acquisition layer, a real-time storage and computing layer, and an AI intelligent analysis layer connected in sequence; The data acquisition layer is used to collect device performance data of the target device, preprocess the device performance data, and send the preprocessed device performance data to the real-time storage and computing layer. The real-time storage and computing layer is used to receive the preprocessed device performance data, perform streaming processing on the preprocessed device performance data to obtain the real-time statistical indicators corresponding to the device performance data, and store the real-time statistical indicators in real time. The AI ​​intelligent analysis layer is used to identify anomalies in the target device based on the real-time statistical indicators and through a dynamically updated Long Short-Term Memory (LSTM) network model to obtain anomaly identification result data corresponding to the target device. If the anomaly identification result data shows that the target device currently has abnormal indicators, the root cause of the abnormal indicators is located to obtain the corresponding root cause location result data.

[0007] In some embodiments of this application, the real-time statistical and intelligent analysis system for device performance data further includes: a visualization and alarm layer; The visualization and alarm layer is used to visualize the real-time statistical indicators and the anomaly identification results data, and to issue anomaly alarms based on the anomaly identification results data and the root cause location results data.

[0008] In some embodiments of this application, the data acquisition layer includes: acquisition agents and edge preprocessing nodes respectively deployed on each of the target devices; The acquisition agent is used to acquire device performance data of the target device according to preset acquisition rules, and transmit the acquired device performance data to the edge preprocessing node via UDP protocol; The edge preprocessing node is used to preprocess the device performance data to obtain preprocessed device performance data, and sends the preprocessed device performance data to the real-time storage and computing layer via Ethernet; wherein, the preprocessing includes at least one of: data cleaning, format standardization, data compression, noise filtering and tag addition.

[0009] In some embodiments of this application, the preset acquisition rules include: an event-triggered mode and a timed replenishment mode; The event triggering mode includes: if the difference between the instantaneous value of the currently monitored device performance data and the value collected at the previous moment exceeds a preset fluctuation threshold, then the collection of the device performance data is triggered; The timed replenishment mode includes: periodically collecting device performance data at preset time intervals when the event triggering mode is not triggered.

[0010] In some embodiments of this application, the real-time storage and computing layer includes: a Flink computing engine and a time-series database; The Flink computing engine is used to receive the preprocessed device performance data, perform streaming processing on the preprocessed device performance data based on a preset streaming processing model to obtain various real-time statistical indicators corresponding to the device performance data and the current actual value of each real-time statistical indicator, and transmit the real-time statistical indicators to the time series database. The time-series database is used to store real-time statistical indicators corresponding to the device performance data in the form of automatic data sharding, so that they can be called by the AI ​​intelligent analysis layer and the Flink computing engine.

[0011] In some embodiments of this application, each of the real-time statistical indicators includes: instantaneous indicators corresponding to the latest data status, sliding window aggregation indicators based on a preset time length, and comparison indicators that are compared with historical data of the same period or data of the previous statistical period.

[0012] In some embodiments of this application, the AI ​​intelligent analysis layer includes: a dynamic model training module, an anomaly recognition module, and a root cause localization module; The dynamic model training module is used to receive the real-time statistical metrics transmitted by the Flink computing engine, and to periodically and automatically update the parameters of the LSTM model based on the real-time statistical metrics. The anomaly identification module is used to perform time-series prediction on each of the real-time statistical indicators using the updated LSTM model to generate the indicator prediction value corresponding to each of the real-time statistical indicators, and calculate the deviation between the indicator prediction value and the actual value corresponding to each of the real-time statistical indicators. If the deviation is greater than a preset threshold, the corresponding real-time statistical indicator is determined to be an abnormal indicator currently existing in the target device. The root cause localization module is used to perform root cause localization of the abnormal indicators based on a knowledge graph when it is determined that the target device currently has the abnormal indicators, so as to obtain the corresponding root cause localization result data; wherein, the knowledge graph is used to store the correspondence between each abnormal indicator and the root cause localization result data, and the root cause localization result data includes: device hardware, software configuration and business load.

[0013] In some embodiments of this application, the visualization and alarm layer includes: a real-time dashboard and an intelligent alarm module; The real-time dashboard is used to visualize the real-time statistical indicators and the anomaly identification results data; The intelligent alarm module is used to determine the current severity of the anomaly of the target device based on the anomaly identification result data and the root cause location result data, according to a preset anomaly severity table, and to push alarm information containing the anomaly identification result data, the root cause location result data, and the anomaly severity; wherein, the anomaly severity table is used to store the correspondence between the identifier of the anomaly indicator, the root cause location result data, and the type of anomaly severity.

[0014] A second aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the functions of each module in the real-time statistical and intelligent analysis system for device performance data.

[0015] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the functions of each module in the real-time statistical and intelligent analysis system for device performance data.

[0016] The fourth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the functions of each module in the real-time statistical and intelligent analysis system for device performance data.

[0017] The real-time statistical and intelligent analysis system for equipment performance data provided in this application includes a data acquisition layer, a real-time storage and computing layer, and an AI intelligent analysis layer connected in sequence. The data acquisition layer collects equipment performance data from a target device, preprocesses the data, and sends the preprocessed data to the real-time storage and computing layer. The real-time storage and computing layer receives the preprocessed data, performs streaming processing on it to obtain real-time statistical indicators corresponding to the data, and stores these indicators in real-time. The AI ​​intelligent analysis layer is used to dynamically update the data based on the real-time statistical indicators. A novel Long Short-Term Memory (LSTM) network model is used to identify anomalies in the target device to obtain anomaly identification result data. If the anomaly identification result data shows that the target device currently has abnormal indicators, the root cause of the abnormal indicators is located to obtain the corresponding root cause location result data. This can solve the problem of high latency in performance data statistics in existing technologies, achieve millisecond-level data acquisition, calculation and result output, and meet the requirements of high real-time scenarios. It can break through the limitations of traditional analysis methods, improve the accuracy of implicit anomaly identification through dynamic learning of AI models and association with multi-dimensional data, and achieve automatic anomaly root cause location. It can also reduce system deployment and maintenance costs and simplify architecture design.

[0018] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0019] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the first architecture of a real-time statistical and intelligent analysis system for device performance data in one embodiment of this application.

[0021] Figure 2 This is a schematic diagram of a second architecture for a real-time statistical and intelligent analysis system for device performance data in one embodiment of this application.

[0022] Figure 3 A schematic diagram of the third architecture of the real-time statistical and intelligent analysis system for device performance data in one embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0024] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0026] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0027] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0028] It should be noted that, in one method for statistical analysis of equipment performance data, the following can be used: (1) Data collection: Deploy a collection agent on the target device to collect performance data such as CPU utilization and memory usage at fixed intervals (5-10 seconds) and transmit it to the data center via TCP protocol; (2) Data storage: Relational databases (such as MySQL) are used to store the collected data. Tables need to be created manually (by date or device number) to alleviate storage pressure. (3) Statistics and analysis: Calculate basic indicators (such as the average CPU utilization rate within 1 minute) using SQL statements and generate static reports using Excel or simple BI tools; anomaly judgment depends on preset thresholds, and only pushes information on indicators exceeding the threshold after triggering an alarm, without further analysis functions.

[0029] However, the method for statistical analysis of the equipment's performance data has the following problems: (1) Insufficient real-time performance: Fixed collection interval (5-10 seconds) and relational database write delay (about 2-3 seconds) cause performance data statistics to lag behind the actual state by 8-13 seconds, which cannot meet the real-time requirements of scenarios such as finance and industry. (2) Weak analytical capabilities: It can only identify obvious anomalies when indicators exceed the threshold, but cannot discover hidden problems (such as the risk of leakage when the daily average memory usage rate increases by 2%); and there is no problem location function, so maintenance personnel need to manually associate multi-dimensional data such as CPU, memory, and network, which takes a long time to investigate. (3) High storage cost: Relational databases have low storage efficiency for massive time-series data. A single server needs to occupy about 15GB of storage space for 10GB of performance data per day. Moreover, when querying historical data, cross-table joins are required, and the response time exceeds 10 seconds.

[0030] In another performance data analysis system based on a big data framework, the following approach can be adopted: (1) Data acquisition: Flume, Kafka and other tools are used to build data transmission channels, the acquisition interval is shortened to 1-2 seconds, and high throughput data reception is supported (100,000 data entries can be processed per second in a single channel). (2) Data storage and statistics: Historical data is stored using HDFS and processed in near real-time using Spark Streaming (with a delay of about 1-2 seconds). It can also be used to calculate complex metrics such as the maximum value of performance metrics within a 5-minute sliding window. (3) Analysis function: The K-means clustering algorithm is introduced to classify historical performance data and identify common abnormal patterns (such as the abnormal combination of sudden increase in CPU and sudden decrease in memory). However, model training requires manual triggering and only outputs abnormal labels without root cause analysis.

[0031] However, this performance data analysis system based on a big data framework has the following problems: (1) Real-time computing latency still exists: Spark Streaming's minimum batch processing interval is 1 second. With the addition of data transmission and computing time, the final statistical results are delayed by about 2-3 seconds, which cannot meet the millisecond-level real-time monitoring scenarios (such as high-frequency trading systems). (2) Poor flexibility of AI models: K-means models need to be retrained manually on a regular basis (e.g., weekly) using historical data, and cannot dynamically adjust parameters according to real-time data changes; (3) Complex deployment and maintenance: It relies on the collaboration of multiple components such as Hadoop and Spark, and requires professional big data operation and maintenance personnel to configure the cluster.

[0032] Therefore, in order to solve the problem of high latency in performance data statistics in the prior art, overcome the limitations of traditional analysis methods, and reduce system deployment and maintenance costs, this application provides a real-time statistical and intelligent analysis system for device performance data, a physical device, a computer-readable storage medium, and a computer program product. It can realize millisecond-level real-time device anomaly performance analysis from data acquisition to result output. By associating AI models with dynamic learning capabilities and multi-dimensional data, it can significantly improve the accuracy of identifying latent anomalies and the efficiency of root cause location. At the same time, the system architecture is simplified, which can effectively reduce deployment and maintenance costs.

[0033] The following examples will provide a detailed description.

[0034] Based on this, embodiments of this application provide a real-time statistical and intelligent analysis system for equipment performance data, see [link to relevant documentation]. Figure 1 The real-time statistical and intelligent analysis system for equipment performance data specifically includes the following: The data acquisition layer 1, real-time storage and computing layer 2, and AI intelligent analysis layer 3 are connected sequentially.

[0035] AI (Artificial Intelligence) refers to the technology of simulating human intelligence through computer programs, which can realize complex tasks such as data learning, reasoning, and anomaly recognition. In this system, it is used for in-depth analysis of performance data and problem localization.

[0036] The data acquisition layer 1 is used to collect device performance data of the target device, preprocess the device performance data, and send the preprocessed device performance data to the real-time storage and computing layer 2.

[0037] In one example, referring to Table 1, the device performance data may include at least one of the following: hardware resource data of the target device, software and application data, business and user experience data, and domain-specific data, and each type of data contains multiple subcategories of data.

[0038] Table 1 The real-time storage and computing layer 2 is used to receive the preprocessed device performance data, perform streaming processing on the preprocessed device performance data to obtain the real-time statistical indicators corresponding to the device performance data, and store the real-time statistical indicators in real time.

[0039] Streaming processing (also known as real-time statistics) refers to the collection, calculation and summarization of system performance data (such as CPU utilization and response time) at the millisecond level, ensuring that the time difference between the data results and the actual operating status does not exceed 1 second, thus meeting the requirements of real-time monitoring.

[0040] The AI ​​intelligent analysis layer 3 is used to identify anomalies in the target device based on the real-time statistical indicators and through a dynamically updated Long Short-Term Memory (LSTM) network model to obtain anomaly identification result data corresponding to the target device. If the anomaly identification result data shows that the target device currently has anomalies, then the root cause of the anomalies is located to obtain the corresponding root cause location result data.

[0041] It should be noted that the device or target device mentioned in this application may refer to computer equipment, and the device performance data may refer to key indicator data used to reflect the operating status of the target system (such as server, application), including but not limited to hardware resource utilization, data transmission rate, business request success rate, etc.

[0042] It is understood that the number of root cause location results may include information such as the device hardware, software configuration, and business load involved in the abnormal indicators, and may also include: (1) Propagation path analysis: Demonstrates the propagation path of the fault in the system topology. For example: increased network latency, application service response timeouts, thread pool blocking leading to a surge in application server CPU usage; (2) Temporal correlation: Prove the chronological relationship between the root cause event and the anomaly. For example: Two minutes after the configuration release event (occurring at 14:05), the application service error rate began to rise; (3) Impact Scope Assessment: Indicate the scope of business or systems affected by this root cause issue. For example: Affected business: order creation, payment process; Affected users: users in the Asian region.

[0043] As can be seen from the above description, the real-time statistics and intelligent analysis system for device performance data provided in this application can solve the problem of high latency in performance data statistics in the prior art, realize millisecond-level data acquisition, calculation and result output, and meet the requirements of high real-time scenarios; it can break through the limitations of traditional analysis methods, improve the accuracy of hidden anomaly identification through dynamic learning of AI models and association with multi-dimensional data, and realize automatic location of anomaly root causes; it can reduce system deployment and maintenance costs and simplify architecture design.

[0044] To efficiently and intuitively present the massive amounts of real-time data, complex anomaly information, and root cause analysis conclusions generated by the system to operations and maintenance personnel, and to ensure that key information can be perceived and responded to in a timely and accurate manner, thereby transforming the results of technical analysis into practical operations and maintenance decisions and actions, this application provides a real-time statistical and intelligent analysis system for equipment performance data. (See also...) Figure 2 The real-time statistics and intelligent analysis system for equipment performance data also specifically includes the following: Visualization and Alarming Layer 4.

[0045] The visualization and alarm layer 4 is used to visualize the real-time statistical indicators and the anomaly identification results data, and to issue anomaly alarms based on the anomaly identification results data and the root cause location results data.

[0046] In other words, the system provided in this embodiment adopts a four-layer architecture design, including a data acquisition layer, a real-time storage and computing layer, an AI intelligent analysis layer, and a visualization and alarm layer. Each layer works together through standardized interfaces.

[0047] As can be seen from the above description, the real-time statistical and intelligent analysis system for equipment performance data provided in this application embodiment breaks down data silos by comprehensively displaying real-time statistical indicators, anomaly identification results, and root cause location data on the same visual interface (such as dashboard or topology diagram). This enables operation and maintenance personnel to intuitively and quickly establish an understanding of the overall system status, greatly shortening the information understanding and decision-making path.

[0048] To further improve the effectiveness and reliability of data acquisition, in a real-time statistical and intelligent analysis system for device performance data provided in this application embodiment, see [link to relevant documentation]. Figure 3 The data acquisition layer 1 of the real-time statistics and intelligent analysis system for equipment performance data specifically includes the following: The acquisition agent 11 and the edge preprocessing node 12 are respectively deployed on each of the target devices.

[0049] The acquisition agent 11 is used to acquire device performance data of the target device according to preset acquisition rules, and transmit the acquired device performance data to the edge preprocessing node 12 via UDP protocol.

[0050] The edge preprocessing node 12 is used to preprocess the device performance data to obtain preprocessed device performance data, and send the preprocessed device performance data to the real-time storage and computing layer 2 via Ethernet; wherein, the preprocessing includes at least one of: data cleaning, format standardization, data compression, noise filtering and tag addition.

[0051] Specifically, edge preprocessing nodes can be implemented using devices such as edge servers for data preprocessing. The acquisition agent is a lightweight software program running on the target device; its core function is to collect raw performance data from the device's operating system or monitoring interface according to preset rules and send the data to the edge preprocessing node.

[0052] In one example, the preprocessing procedure includes: (1) Data reception and parsing: The edge preprocessing node 12 continuously receives data packets sent by the acquisition agent via UDP protocol through the Socket listening port. Then, it unpacks the data packets and parses the original performance data and metadata (such as device ID, timestamp, and metric name) according to the predefined protocol format (such as JSON, Protobuf).

[0053] (2) Data cleaning and validation: Responsible for removing invalid and dirty data to ensure data quality. This may include: Invalid value filtering: Identify and discard data points that are obviously illogical, such as negative CPU utilization (-1%) or memory usage exceeding the total physical memory. In addition, format validation and repair: check whether data fields are complete and whether the format conforms to specifications. For minor format errors (such as inconsistent timestamp formats), attempt automatic repair and standardization.

[0054] (3) Noise filtering and preliminary smoothing: Lightweight noise filtering algorithms (such as outlier detection based on Z-Score or simple rate of change thresholding) are deployed at edge nodes to identify and filter out abnormal spikes that are extremely short in duration due to instantaneous system jitter. This can effectively prevent false alarms from subsequent AI analysis layers and improve analysis accuracy.

[0055] (4) Data formatting and standardization: This involves converting device performance data from different manufacturers, models, or operating systems into a unified internal standard data model. This may include: Standardize field names: For example, unify "cpu_usage" and "cpu.util" into "cpu_usage_percent"; Unit standardization: For example, converting memory usage from KB and MB to GB; In addition, timestamp alignment: align the timestamps of all data to the standard time zone and format them in a uniform format.

[0056] (5) Data enrichment: Add valuable contextual tags to data records, such as device tags: add static information such as device model, computer room, rack location, and business department; or topology tags: add information about the role of the device in the business architecture, such as: Web server, database master node, message queue cluster.

[0057] (6) Data Compression and Batch Aggregation: To optimize network transmission efficiency, the processed data is compressed and aggregated. Compression can employ efficient lossless or lossy compression algorithms (such as GZIP and Snappy) to reduce network bandwidth usage. Micro-batch aggregation can aggregate multiple data records into a single batch in memory and then send it all at once to the real-time storage and computing layer 2, thereby reducing the number of network connections and increasing throughput.

[0058] In order to further improve the timeliness of collecting device performance data of target devices and reduce the resource occupancy rate of collection, in the device performance data real-time statistics and intelligent analysis system provided in this application embodiment, the preset collection rules specifically include: event trigger mode and timed supplementation mode; The event triggering mode includes: if the difference between the instantaneous value of the currently monitored device performance data and the value collected at the previous moment exceeds a preset fluctuation threshold, then the collection of the device performance data is triggered; The timed replenishment mode includes: periodically collecting device performance data at preset time intervals when the event triggering mode is not triggered.

[0059] Specifically, the data acquisition layer consists of a lightweight acquisition agent and edge preprocessing nodes. One acquisition agent is deployed on each target device. The acquisition agent transmits raw data to the edge preprocessing nodes via the UDP protocol (low latency). The edge nodes are connected to the real-time storage and computing layer via Ethernet. The acquisition mode can adopt both event-triggered and timed supplementation modes. That is, when the performance data fluctuation exceeds a preset small threshold, acquisition is triggered immediately. When there is no fluctuation, data is acquired at 500ms intervals to ensure data integrity and real-time performance.

[0060] To further improve the effectiveness and reliability of real-time storage and computing, in a real-time statistical and intelligent analysis system for device performance data provided in this application embodiment, see [link to relevant documentation]. Figure 3 The real-time storage and computing layer 2 of the real-time statistics and intelligent analysis system for device performance data specifically includes the following: Flink computing engine 21 and time series database 22; The Flink computing engine 21 is used to receive the preprocessed device performance data, perform streaming processing on the preprocessed device performance data based on a preset streaming processing model to obtain various real-time statistical indicators corresponding to the device performance data and the current actual value of each real-time statistical indicator, and transmit the real-time statistical indicators to the time series database 22. The time-series database 22 is used to store real-time statistical indicators corresponding to the device performance data in the form of automatic data sharding, so that they can be called by the AI ​​intelligent analysis layer 3 and the Flink computing engine 21.

[0061] Among them, time-series databases are storage systems designed specifically for time-series data (such as continuously generated performance data). They feature high write speed and low query latency, and can efficiently store and retrieve massive amounts of historical and real-time performance data.

[0062] Specifically, the real-time storage and computing layer can consist of a time-series database (InfluxDB, supporting 100,000 data entries per second) and a real-time computing engine (Flink, with stream processing latency ≤100ms). Edge preprocessing nodes transmit preprocessed data to the Flink computing engine, and Flink's computation results are written to InfluxDB in real time. Simultaneously, InfluxDB provides historical data query support for Flink. Flink, based on a stream processing model, calculates multi-dimensional metrics in real time, including instantaneous values, sliding window values, and year-on-year / month-on-month values, with a computation result latency ≤300ms. For efficient storage, InfluxDB indexes data by time and device ID, supports automatic data sharding, and deletes expired data.

[0063] To further improve the effectiveness and reliability of real-time statistical indicators, in a real-time statistical and intelligent analysis system for equipment performance data provided in this application embodiment, each of the real-time statistical indicators includes: instantaneous indicators corresponding to the latest data status, sliding window aggregation indicators based on a preset time length, and comparison indicators that are compared with historical data of the same period or data of the previous statistical period.

[0064] Specifically, instantaneous indicators refer to the most recently received or directly calculated values ​​based on the latest data, reflecting the system's state at the most recent data collection point. Instantaneous indicators include: (1) Instantaneous values ​​of hardware resources: instantaneous values ​​of CPU utilization, memory usage, disk I / O rate, and network receive / send traffic.

[0065] (2) Software and application instantaneous values: number of currently active processes and instantaneous value of application service response time (referring to the time taken for a single request).

[0066] (3) Business instantaneous value: The processing result (success / failure) of a single business request.

[0067] The sliding window aggregation metric refers to the statistical value obtained by real-time aggregation calculation of all performance data flowing into a continuously sliding, fixed-length time window (such as the most recent 5 minutes or 1 minute). The sliding window aggregation metric includes: (1) Central tendency indicators: arithmetic mean and median of all data points within the window.

[0068] (2) Extreme values ​​and range indicators: the maximum and minimum values ​​of all data points within the window.

[0069] (3) Distribution indicators: P95 quantile and P99 quantile of all data points in the window (which can effectively reflect tail delay and are crucial to user experience).

[0070] (4) Volatility indicators: standard deviation and variance of all data points within the window (reflecting the stability of the system within that time period).

[0071] (5) Rate and counting metrics: Requests per second (QPS / TPS), number of errors per unit time.

[0072] Among them, the comparative indicator refers to a derived indicator calculated by comparing the indicator value of the current statistical period with a selected historical benchmark value. It is used to quantify the trend and degree of anomaly in performance. The comparative indicators include: (1) Month-on-month change rate: This compares the aggregated index value of the current time window with the index value of the previous time window of the same length. The calculation formula is: (current value - value of the previous period) / value of the previous period. This is used to detect sudden increases or decreases in the index.

[0073] (2) Year-on-year change rate: This compares the indicator value at the current moment or within the current time window with the indicator value for the same period in history (e.g., the same moment yesterday, the same moment last week). The calculation formula is similar to that for month-on-month comparison. It is used to identify anomalies caused by changes in cyclical business models or specific events.

[0074] To further improve the effectiveness and reliability of AI intelligent analysis, in a real-time statistical and intelligent analysis system for device performance data provided in this application embodiment, see [link to relevant documentation]. Figure 3The AI ​​intelligent analysis layer 3 in the real-time statistics and intelligent analysis system for device performance data specifically includes the following: Dynamic model training module 31, anomaly identification module 32, and root cause localization module 33; The dynamic model training module 31 is used to receive the real-time statistical indicators transmitted by the Flink computing engine 21, and to periodically and automatically update the parameters of the LSTM model based on the real-time statistical indicators. The anomaly identification module 32 is used to perform time-series prediction on each of the real-time statistical indicators using the updated LSTM model to generate the indicator prediction value corresponding to each of the real-time statistical indicators, and calculate the deviation between the indicator prediction value and the actual value corresponding to each of the real-time statistical indicators. If the deviation is greater than a preset threshold, the corresponding real-time statistical indicator is determined to be an abnormal indicator currently existing in the target device. The root cause localization module 33 is used to perform root cause localization of the abnormal indicators based on a knowledge graph when it is determined that the target device currently has the abnormal indicators, in order to obtain corresponding root cause localization result data. The knowledge graph stores the correspondence between each abnormal indicator and the root cause localization result data, and the root cause localization result data includes: device hardware, software configuration, and service load. That is, each node in the knowledge graph represents a different abnormal indicator, device hardware, software configuration, and service load; the edges between nodes represent the relationships between nodes.

[0075] Specifically, the AI ​​intelligent analysis layer consists of a dynamic model training module, an anomaly detection module, and a root cause localization module. The Flink computing engine transmits real-time indicator data to the AI ​​intelligent analysis layer. The dynamic model training module and the anomaly detection module interact bidirectionally (model parameters are updated in real time), and the anomaly detection module transmits the anomaly results to the root cause localization module. The dynamic model training uses the LSTM (Long Short-Term Memory) algorithm, receives indicator data transmitted by Flink in real time, automatically updates model parameters hourly, learns new performance data patterns, and adapts to changes in system operating status. The LSTM model predicts real-time indicator data, and if the deviation between the actual value and the predicted value exceeds a preset threshold, it is judged as an anomaly. Based on knowledge graph technology, it associates anomaly indicators with multi-dimensional data such as device hardware, software configuration, and business load, and automatically generates root cause conclusions.

[0076] In one example, a knowledge graph consists of the following core elements: 1. Entities (Nodes): Entities represent specific objects in the system environment and are the core building blocks of a knowledge graph. They mainly include: (1) Hardware entities: such as servers, CPUs, memory modules, disks, network switches, and routers. Their attributes may include model, capacity, specifications, etc.

[0077] (2) Software entities: such as operating systems, applications, microservices, databases, middleware, and configuration files. Their attributes may include version number, startup parameters, listening port, etc.

[0078] (3) Business entities: such as business transactions, user sessions, front-end load balancers, and API gateways. Their attributes may include expected throughput, criticality (SLA level), etc.

[0079] (4) Metric entities: such as CPU utilization, API response time, and error rate. This entity is the bridge connecting the exception and its root cause.

[0080] 2. Relationships (edges): Relationships define the ways in which entities interact, depend on, and influence each other; they are the logical chains for root cause reasoning. They mainly include: (1) Topology: describes physical or logical connections. For example: "Server - running on → application, application - dependent on → database, microservice A - calling → microservice B".

[0081] (2) Causal and Influence Relationships: This is the core relationship for achieving intelligent positioning. It describes how a change in the state of one entity leads to a change in the state of another entity. These relationships can be learned through historical data analysis, operational experience rules, or AI models. For example: "High disk I / O latency - leads to slow database queries, memory leaks - leads to application crashes, network packet loss - affects API timeouts."

[0082] (3) Subordination: such as “CPU - belongs to → server, configuration file - configured in → application”.

[0083] 3. Attributes (Features of nodes and edges): Attributes describe the specific characteristics and states of an entity or relationship in the form of key-value pairs. These mainly include: (1) Entity attributes: For example, the server entity has attributes such as IP address, computer room location, and maintenance manager.

[0084] (2) Relationship attributes: such as confidence level (0.95), average impact delay (500ms), etc., which are used to quantify and sort among multiple possible root causes.

[0085] To further improve the effectiveness and reliability of visualization and alarms, in a real-time statistical and intelligent analysis system for device performance data provided in this application embodiment, see [link to relevant documentation]. Figure 3 The visualization and alarm layer 4 of the real-time statistics and intelligent analysis system for device performance data specifically includes the following: Real-time dashboard 41 and intelligent alarm module 42; The real-time dashboard 41 is used to visualize the real-time statistical indicators and the anomaly identification results data; The intelligent alarm module 42 is used to determine the current severity of the anomaly of the target device based on the anomaly identification result data and the root cause location result data, according to a preset anomaly severity table, and to push alarm information containing the anomaly identification result data, the root cause location result data, and the anomaly severity; wherein, the anomaly severity table is used to store the correspondence between the identifier of the anomaly indicator, the root cause location result data, and the type of anomaly severity.

[0086] Specifically, the visualization and alerting layer consists of a real-time dashboard (accessible via web / mobile) and an intelligent alerting module. InfluxDB transmits historical and real-time metric data to the real-time dashboard, while the AI ​​intelligent analysis layer transmits anomaly information and root cause conclusions to the intelligent alerting module. The alerting module and the dashboard work in tandem. Real-time visualization displays performance metric trends in the form of line charts, heatmaps, and dashboards. Intelligent alerts push alert information via SMS, email, WeChat Work, and other channels based on the severity of the anomaly (general / severe / urgent).

[0087] In another embodiment of this application, the data acquisition layer can replace the "lightweight Agent and edge nodes" with "cloud-native acquisition components (such as PrometheusExporter)" to directly connect to the cloud platform; this is suitable for target systems deployed entirely in the cloud and can reduce local hardware investment; however, it has poor compatibility with local servers deployed in the cloud and requires additional development of adaptation plugins. The AI ​​intelligent analysis layer can also replace the LSTM algorithm with the XGBoost (Extreme Gradient Boosting Tree) algorithm to build a classification model based on historical performance data and identify anomaly types; the real-time storage and computing layer can also replace InfluxDB with TimescaleDB. TimescaleDB supports SQL query syntax, so operators do not need to learn new syntax and it has good compatibility with traditional relational databases; however, in high-concurrency write scenarios, the write latency is higher than that of InfluxDB.

[0088] In other words, the real-time statistical and intelligent analysis system for device performance data provided in this application embodiment designs a performance data acquisition method based on event triggering and timed supplementation, including acquisition threshold setting rules for acquisition agents, preprocessing procedures for edge nodes, and data transmission protocol selection (UDP); a real-time performance data calculation and storage system, including a collaborative connection structure between the Flink computing engine and a time-series database, and real-time calculation logic for multi-dimensional indicators; and a performance anomaly analysis method based on LSTM and knowledge graphs, including a dynamic update process for the LSTM model, a deviation threshold setting method for anomaly identification, and root cause localization rules for multi-dimensional data association.

[0089] To further illustrate the above embodiments, this application also provides an application example of a real-time device performance data statistics and intelligent analysis system, which can also be called an artificial intelligence-driven real-time performance data statistics and intelligent analysis system. Taking the monitoring of web servers in small and medium-sized Internet enterprises as an example, the system deployment and operation process is as follows: (1) Target system: 10 Web servers (equipped with Linux system, running e-commerce website backend services); (2) Hardware configuration: edge preprocessing node (1 unit, CPU 4 cores, memory 8GB), real-time storage and computing node (1 unit, CPU 8 cores, memory 16GB), AI analysis and visualization node (1 unit, CPU 8 cores, memory 32GB).

[0090] (3) Data acquisition layer deployment: Lightweight Agent is installed on 10 Web servers respectively, and the event trigger threshold is configured as follows: CPU utilization change ≥2%, memory usage change ≥1%, and the timed replenishment interval is 500ms; the edge preprocessing node is deployed locally on the server cluster to receive the data transmitted by the Agent, clean it (filter invalid values ​​such as "CPU utilization = -1%)" and unify the format, and the processed data is about 1000 records per second.

[0091] (4) Real-time storage and computing layer: The Flink computing engine receives edge node data in real time and calculates 10 types of indicators such as instantaneous CPU utilization, maximum memory in a 5-minute sliding window, and year-on-year change rate of response time. The computing latency is about 200ms. The computing results are written to InfluxDB in real time, and the data retention period is configured to be 30 days. The average daily data storage of a single server is about 5GB.

[0092] (5) AI intelligent analysis layer: The LSTM model is initially trained using historical performance data of a company over the past 3 months, and the parameters are updated hourly using real-time data after deployment.

[0093] (6) Visualization and Alarm Layer: Real-time dashboard displays the server's memory usage trend and marks abnormal time periods.

[0094] In summary, the real-time equipment performance data statistics and intelligent analysis system provided in this application example has the following features: 1. Dual-mode data collection technology with event triggering and timed replenishment: By dynamically adjusting the collection timing, the agent resource consumption is reduced while ensuring data real-time performance, thus resolving the contradiction between real-time collection and resource consumption. 2. Real-time computing architecture that integrates Flink with time-series databases: The Flink stream processing engine directly interfaces with time-series databases to achieve integrated computing and storage, avoiding data transfer delays; 3. Analysis technique that integrates LSTM dynamic model and knowledge graph: The LSTM model automatically updates its parameters every hour to adapt to changes in system state; the knowledge graph associates multi-dimensional data to enable rapid location of the root cause of anomalies.

[0095] In other words, this application addresses the high latency issue in performance data statistics in existing technologies, achieving millisecond-level data acquisition, computation, and result output to meet the demands of high real-time scenarios. It overcomes the limitations of traditional analysis methods by using AI models for dynamic learning and multi-dimensional data correlation to improve the accuracy of hidden anomaly identification and automatically locate the root cause of anomalies. It also reduces system deployment and maintenance costs and simplifies architecture design. The analytical capabilities are significantly enhanced; the LSTM dynamic model effectively improves anomaly identification accuracy, and knowledge graph technology enables root cause location within seconds, greatly improving operational and troubleshooting efficiency.

[0096] The embodiments of the real-time statistics and intelligent analysis device for equipment performance data provided in this application can be used to execute the processing flow of the embodiments of the real-time statistics and intelligent analysis system for equipment performance data in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the real-time statistics and intelligent analysis system for equipment performance data in the above embodiments.

[0097] The real-time statistics and intelligent analysis of device performance data in the aforementioned device performance data real-time statistics and intelligent analysis device can be performed on the client device. Specifically, the selection can be based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed on the client device, the client device may further include a processor for the specific processing of real-time statistics and intelligent analysis of device performance data.

[0098] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0099] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the aforementioned protocols.

[0100] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the real-time statistical and intelligent analysis system for device performance data mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0101] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0102] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the real-time device performance data statistics and intelligent analysis system in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby realizing the real-time device performance data statistics and intelligent analysis system in the above method embodiments.

[0103] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The one or more modules are stored in the memory, and when executed by the processor, they form a real-time statistical and intelligent analysis system for device performance data in the execution embodiment.

[0105] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0106] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0107] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0108] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned real-time statistical and intelligent analysis system for device performance data. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0109] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned real-time statistical and intelligent analysis system for device performance data.

[0110] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0111] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0112] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0113] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A real-time statistical and intelligent analysis system for equipment performance data, characterized in that, include: The data acquisition layer, real-time storage and computing layer, and AI intelligent analysis layer are connected sequentially. The data acquisition layer is used to collect device performance data of the target device, preprocess the device performance data, and send the preprocessed device performance data to the real-time storage and computing layer. The real-time storage and computing layer is used to receive the preprocessed device performance data, perform streaming processing on the preprocessed device performance data to obtain the real-time statistical indicators corresponding to the device performance data, and store the real-time statistical indicators in real time. The AI ​​intelligent analysis layer is used to identify anomalies in the target device based on the real-time statistical indicators and through a dynamically updated Long Short-Term Memory (LSTM) network model to obtain anomaly identification result data corresponding to the target device. If the anomaly identification result data shows that the target device currently has abnormal indicators, the root cause of the abnormal indicators is located to obtain the corresponding root cause location result data.

2. The real-time statistical and intelligent analysis system for equipment performance data according to claim 1, characterized in that, Also includes: Visualization and alerting layer; The visualization and alarm layer is used to visualize the real-time statistical indicators and the anomaly identification results data, and to issue anomaly alarms based on the anomaly identification results data and the root cause location results data.

3. The real-time statistical and intelligent analysis system for equipment performance data according to claim 1, characterized in that, The data acquisition layer includes: acquisition agents and edge preprocessing nodes deployed on each of the target devices; The acquisition agent is used to acquire device performance data of the target device according to preset acquisition rules, and transmit the acquired device performance data to the edge preprocessing node via UDP protocol; The edge preprocessing node is used to preprocess the device performance data to obtain preprocessed device performance data, and sends the preprocessed device performance data to the real-time storage and computing layer via Ethernet; wherein, the preprocessing includes at least one of: data cleaning, format standardization, data compression, noise filtering and tag addition.

4. The real-time statistical and intelligent analysis system for equipment performance data according to claim 3, characterized in that, The preset data collection rules include: event-triggered mode and timed replenishment mode; The event triggering mode includes: if the difference between the instantaneous value of the currently monitored device performance data and the value collected at the previous moment exceeds a preset fluctuation threshold, then the collection of the device performance data is triggered; The timed replenishment mode includes: periodically collecting device performance data at preset time intervals when the event triggering mode is not triggered.

5. The real-time statistical and intelligent analysis system for equipment performance data according to claim 1, characterized in that, The real-time storage and computing layer includes: the Flink computing engine and a time-series database; The Flink computing engine is used to receive the preprocessed device performance data, perform streaming processing on the preprocessed device performance data based on a preset streaming processing model to obtain various real-time statistical indicators corresponding to the device performance data and the current actual value of each real-time statistical indicator, and transmit the real-time statistical indicators to the time series database. The time-series database is used to store real-time statistical indicators corresponding to the device performance data in the form of automatic data sharding, so that they can be called by the AI ​​intelligent analysis layer and the Flink computing engine.

6. The real-time statistical and intelligent analysis system for equipment performance data according to claim 1, characterized in that, Each of the aforementioned real-time statistical indicators includes: instantaneous indicators corresponding to the latest data status, sliding window aggregation indicators based on a preset time length, and comparison indicators that are compared with historical data from the same period or data from the previous statistical period.

7. The real-time statistical and intelligent analysis system for equipment performance data according to claim 5, characterized in that, The AI ​​intelligent analysis layer includes: a dynamic model training module, an anomaly recognition module, and a root cause localization module; The dynamic model training module is used to receive the real-time statistical metrics transmitted by the Flink computing engine, and to periodically and automatically update the parameters of the LSTM model based on the real-time statistical metrics. The anomaly identification module is used to perform time-series prediction on each of the real-time statistical indicators using the updated LSTM model to generate the indicator prediction value corresponding to each of the real-time statistical indicators, and calculate the deviation between the indicator prediction value and the actual value corresponding to each of the real-time statistical indicators. If the deviation is greater than a preset threshold, the corresponding real-time statistical indicator is determined to be an abnormal indicator currently existing in the target device. The root cause localization module is used to perform root cause localization of the abnormal indicators based on a knowledge graph when it is determined that the target device currently has the abnormal indicators, so as to obtain the corresponding root cause localization result data; wherein, the knowledge graph is used to store the correspondence between each abnormal indicator and the root cause localization result data, and the root cause localization result data includes: device hardware, software configuration and business load.

8. The real-time statistical and intelligent analysis system for equipment performance data according to claim 2, characterized in that, The visualization and alarm layer includes: a real-time dashboard and an intelligent alarm module; The real-time dashboard is used to visualize the real-time statistical indicators and the anomaly identification results data; The intelligent alarm module is used to determine the current severity of the anomaly of the target device based on the anomaly identification result data and the root cause location result data, according to a preset anomaly severity table, and to push alarm information containing the anomaly identification result data, the root cause location result data, and the anomaly severity; wherein, the anomaly severity table is used to store the correspondence between the identifier of the anomaly indicator, the root cause location result data, and the type of anomaly severity.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the functions of the real-time statistical and intelligent analysis system for device performance data as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the functions of the real-time statistical and intelligent analysis system for device performance data as described in any one of claims 1 to 8.

Images