Monitoring data processing method and device, computer equipment, medium and program product

Abstract

The invention relates to a monitoring data processing method and device, computer equipment, a medium and a program product. The method comprises the following steps: acquiring real-time system monitoring data of a target server; constructing the real-time system monitoring data into a distributed sequence index structure to obtain structured data, and performing data fragmentation on the structured data according to the data volume and timestamp of each data item in the structured data to obtain target analysis data; and extracting data features in the target analysis data, generating a data monitoring chart according to the data features, and distributing the data monitoring chart to a corresponding client. By adopting the method, the monitoring data processing efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a monitoring data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of Internet technology, Linux systems have been widely deployed in enterprise applications, and the importance of system monitoring has become increasingly prominent.

[0003] Currently, Linux system monitoring mainly relies on data acquisition tools such as SAR (System Activity Reporter) and scheduled tasks for data collection. Data transmission and storage mostly use file synchronization tools such as RSYNC (Remote Synchronization) or SCP (Secure Copy Protocol). Graphical display and PDF (Portable Document Format) document output rely on third-party charting tools and image processing software.

[0004] However, when processing large-scale monitoring data, the data collection and storage methods of these technologies still require manual intervention due to the ever-increasing volume of data and the diverse needs for visualization charts. They cannot adapt to the distributed storage and rapid retrieval visualization requirements of massive monitoring data, and are unable to meet the business needs of real-time monitoring and rapid response. The large amount of data and the diverse types of monitoring reports reduce the efficiency of traditional methods in processing monitoring data. Summary of the Invention

[0005] Therefore, it is necessary to provide a monitoring data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of monitoring data processing in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a monitoring data processing method, including:

[0007] Obtain real-time system monitoring data from the target server;

[0008] The real-time system monitoring data is constructed into a distributed sequence index structure to obtain structured data. Based on the data volume and timestamp of each data item in the structured data, the structured data is fragmented to obtain target analysis data.

[0009] Extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0010] In one embodiment, generating the data monitoring chart based on the data features includes:

[0011] Based on the data characteristics, the data type, numerical distribution, and time series data of the target analysis data are determined, and the chart type is selected according to the data type, the numerical distribution, and the time series data; the chart type includes at least one of line chart, bar chart, area chart, scatter plot, and pie chart;

[0012] Based on the selected chart type, generate the corresponding chart configuration parameters to obtain the data monitoring chart.

[0013] In one embodiment, distributing the data monitoring chart to the corresponding client includes:

[0014] Select a data report template based on the chart type and data type corresponding to the data monitoring chart;

[0015] Configure the data source index according to the data report template, and generate a data report based on the data report template, the data source index, and the data monitoring chart;

[0016] Based on the data type, the distribution permission is determined, and based on the distribution permission, the data report is distributed to the corresponding client.

[0017] In one embodiment, the step of segmenting the structured data according to the data volume and timestamp of each data item in the structured data to obtain target analysis data includes:

[0018] Based on the timestamp and the amount of data, calculate the data query frequency and data growth rate of each data item;

[0019] Calculate the data capacity of a single shard based on the data volume and the data growth rate;

[0020] Based on the data query frequency and the data volume, the structured data is sharded according to at least one of the following dimensions: time dimension, server level dimension, or data type dimension, to obtain the target analysis data.

[0021] In one embodiment, the method further includes:

[0022] Obtain the data volume growth monitoring data and load monitoring data of each segment in the target analysis data;

[0023] If the data volume of the fragment exceeds a preset upper limit, the data of the fragment will be allocated to at least one extended fragment;

[0024] If the data volume of adjacent fragments of the fragment is lower than a preset lower limit, the fragment and the adjacent fragments are merged.

[0025] In one embodiment, allocating the data of the shard to at least one extended shard includes:

[0026] Pause the data writing operation on the shard, and divide the data in the shard into multiple data subsets according to the timestamp and / or the corresponding hash value of the data in the shard;

[0027] Create a new extended shard and migrate the data subset to the corresponding extended shard;

[0028] Update the index mapping relationship between the shard and the extended shard, and resume data write operations.

[0029] Secondly, this application also provides a monitoring data processing device, comprising:

[0030] The data acquisition module is used to acquire real-time system monitoring data from the target server.

[0031] The data processing module is used to construct the real-time system monitoring data into a distributed sequence index structure to obtain structured data, and to perform data fragmentation on the structured data according to the data volume and timestamp of each data item in the structured data to obtain target analysis data.

[0032] The data distribution module is used to extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0033] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0034] Obtain real-time system monitoring data from the target server;

[0035] The real-time system monitoring data is constructed into a distributed sequence index structure to obtain structured data. Based on the data volume and timestamp of each data item in the structured data, the structured data is fragmented to obtain target analysis data.

[0036] Extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0038] Obtain real-time system monitoring data from the target server;

[0039] The real-time system monitoring data is constructed into a distributed sequence index structure to obtain structured data. Based on the data volume and timestamp of each data item in the structured data, the structured data is fragmented to obtain target analysis data.

[0040] Extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0041] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0042] Obtain real-time system monitoring data from the target server;

[0043] The real-time system monitoring data is constructed into a distributed sequence index structure to obtain structured data. Based on the data volume and timestamp of each data item in the structured data, the structured data is fragmented to obtain target analysis data.

[0044] Extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0045] The aforementioned monitoring data processing methods, devices, computer equipment, computer-readable storage media, and computer program products acquire real-time system monitoring data from the target server, construct a distributed sequence index structure from the real-time system monitoring data to obtain structured data, and perform data fragmentation on the structured data according to the data volume and timestamp of each data item in the structured data to obtain target analysis data. This achieves intelligent partition management of data, considering both the balance of data storage capacity and ensuring the logical continuity of time-series data, thereby optimizing the performance of distributed storage and the efficiency of data access. Data features are extracted from the target analysis data, and data monitoring charts are generated based on these features, achieving intelligent automatic selection of chart types. The data monitoring charts are then distributed to the corresponding clients, realizing automated transmission and distribution of monitoring results and improving the efficiency of monitoring data processing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an application environment diagram of a monitoring data processing method in one embodiment;

[0048] Figure 2 This is a flowchart illustrating a monitoring data processing method in one embodiment;

[0049] Figure 3 This is a flowchart illustrating step S206 of the monitoring data processing method in one embodiment;

[0050] Figure 4 This is a structural block diagram of a monitoring data processing device in one embodiment;

[0051] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0054] The monitoring data processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network to obtain and display data processed by server 104. A data storage system stores the data processed by server 104. This system can be integrated onto server 104 or located in the cloud or on other network servers. The data storage system can store real-time system monitoring data and other data. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0055] In one exemplary embodiment, such as Figure 2 As shown, a monitoring data processing method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S206. Wherein:

[0056] Step S202: Obtain real-time system monitoring data of the target server.

[0057] Real-time system monitoring data refers to server operating status information collected at a frequency of seconds or minutes through system monitoring components such as SAR tools. It can include CPU (Central Processing Unit) utilization data, memory usage, disk I / O (Input / Output) performance indicators, network traffic statistics, and average load, among other performance indicators.

[0058] For example, server 104 can be deployed in the same or a different server cluster as the target server. It can establish a monitoring data acquisition connection with the target Linux server group and achieve real-time data acquisition through a monitoring agent program deployed on each target server. Server 104 can call the system monitoring toolkit installed on the target server. This toolkit may contain monitoring components such as SAR, IOSTAT (Input / Output Statistics), and MPSTAT (Multi-Processor Statistics), and can continuously collect the server's operating indicators at one-second intervals.

[0059] During data acquisition, server 104 can obtain detailed breakdown data of CPU utilization, including the percentage of user-mode CPU usage, system-mode CPU usage, idle CPU usage, and the percentage of CPU time spent waiting for I / O operations. Server 104 can also collect comprehensive information on memory usage, covering total system memory capacity, currently used memory, cache memory usage, and swap partition usage. Furthermore, server 104 can obtain real-time disk I / O performance metrics, including statistics on the number of disk read / write operations, the number of bytes read / written, the average response time of disk operations, and the depth of the I / O request queue.

[0060] In terms of network monitoring, server 104 can collect network interface traffic statistics, including the number of data packets received and sent, the total number of bytes transmitted, the count of network transmission error packets, and network interface utilization information. While collecting raw monitoring data, server 104 can automatically add precise timestamp information and a unique server identifier to each data record, thereby ensuring data traceability and integrity.

[0061] During data acquisition, server 104 can standardize the format of the acquired raw monitoring data, converting monitoring data from different sources and in different formats into a structured JSON (JavaScript Object Notation) format. Server 104 can also perform data integrity verification, checking whether each monitoring record contains the necessary timestamp field, server identifier field, and monitoring indicator value field, marking and processing records with missing fields. Server 104 can also perform numerical range reasonableness checks, setting reasonable numerical boundaries based on the characteristics of different monitoring indicators. For example, CPU utilization should be within the range of zero to one hundred, and memory usage should not exceed the total system memory capacity; abnormal values ​​exceeding reasonable ranges are identified and marked. Through timestamp continuity verification, server 104 can detect anomalies such as time jumps, time reversals, or data loss during data acquisition, ensuring the continuity and reliability of time-series data. Based on the results of the above quality checks, server 104 can promptly identify problems in the data acquisition process, thereby improving the data reliability of the monitoring system.

[0062] Step S204: Construct the real-time system monitoring data into a distributed sequence index structure to obtain structured data. Then, according to the data volume and timestamp of each data item in the structured data, perform data sharding on the structured data to obtain the target analysis data.

[0063] Distributed sequence index structure refers to an index system with distributed storage capabilities built according to the time series characteristics of monitoring data, similar to the time series index in Elasticsearch. Data sharding refers to a method of dividing a large-scale dataset into multiple relatively independent data subsets according to specific rules. Data can be sharded by time dimension (day or hour), by server level dimension (store server data of different importance separately), or by data type dimension (manage different types of metrics such as CPU monitoring data and memory monitoring data separately).

[0064] For example, server 104 can arrange monitoring data in an ordered manner according to the time dimension, establishing a mapping relationship between timestamps and data records, enabling time-range-based queries to quickly locate relevant data. During index building, server 104 can create dedicated index branches for different types of monitoring metrics; CPU monitoring data, memory monitoring data, disk I / O monitoring data, and network monitoring data each have independent index paths, thereby improving the query efficiency of specific data types. Server 104 can implement inverted indexes, creating reverse index structures for fields in the monitoring data, thus supporting complex combinations of query conditions and fast data retrieval operations. In a distributed architecture design, server 104 can be configured with multiple index nodes, ensuring good responsiveness even when processing massive amounts of monitoring data through distributed data storage and query load balancing.

[0065] For example, server 104 can calculate the data query frequency and data growth rate of each data item based on timestamps and data volume; calculate the data capacity of a single shard based on data volume and data growth rate; and shard the structured data according to at least one of the following dimensions: time dimension, server level dimension, or data type dimension, based on data query frequency and data volume, to obtain the target analysis data. Server 104 can analyze the storage capacity requirements and access frequency characteristics of each data item, and determine the optimal shard size and number of shards through statistical analysis. In time-dimensional sharding, server 104 can dynamically determine the granularity of time sharding based on the time span and growth rate of the data, using finer sharding granularity for time periods with faster data growth and coarser sharding granularity for time periods with relatively stable data volume. In server-level sharding, server 104 can assess the importance level and monitoring data volume of each server, allocating data from high-importance servers to independent high-performance shards to ensure the access priority of server monitoring data. For data type sharding, Server 104 can place frequently analyzed data types in adjacent shards based on access patterns and analysis needs of different monitoring metrics, reducing the overhead of cross-shard queries. By comprehensively considering multiple factors such as data volume, timestamps, and access frequency, Server 104 can generate an optimized sharding allocation scheme.

[0066] Furthermore, server 104 can also obtain data volume growth monitoring data and load monitoring data of each shard in the target analysis data; when the data volume of a shard exceeds the preset upper limit, it performs a shard expansion operation to allocate the data of the shard to at least one expanded shard; when the data volume of adjacent shards of a shard is lower than the preset lower limit, it merges the shard and the adjacent shards.

[0067] When performing a shard expansion operation, for example, server 104 may pause data write operations on the shard, divide the data in the shard into multiple data subsets according to the timestamp and / or corresponding hash value of the data in the shard; create a new expansion shard and migrate the data subsets to the corresponding expansion shards; update the index mapping relationship between the shard and the expansion shards, and resume data write operations.

[0068] In some embodiments, using a Kafka (distributed stream processing platform) and Elasticsearch architecture, server 104 can employ a message queue-based data sharding implementation. Server 104 can send real-time monitoring data to different topics in the Kafka cluster, creating specialized topic categories based on monitoring data types. For example, the `cpu-metrics` topic stores CPU-related monitoring data, the `memory-metrics` topic stores memory-related data, the `disk-io-metrics` topic stores disk I / O data, and the `network-metrics` topic stores network traffic data. Server 104 can configure intelligent partitioning strategies in Kafka, calculating hash values ​​based on server identifiers to ensure that monitoring data from the same server is always routed to the same partition, guaranteeing data timeliness and consistency. Server 104 can deploy Kafka consumer groups, configuring multiple consumer instances to process data streams from different partitions in parallel, achieving high-throughput data consumption and processing capabilities. Regarding data indexing, server 104 can write consumed monitoring data to the Elasticsearch cluster in real-time, using a date-based index naming strategy, such as the index format `monitoring-2024-01-15`, facilitating data lifecycle management and historical data retrieval. Server 104 can be configured with index templates in Elasticsearch to uniformly define field mapping relationships and analyzer configurations, ensuring that indexes created at different times have a consistent data structure. Through the above steps, Server 104 can achieve distributed data storage and redundancy protection, so that the system can still provide normal data access services when a single node fails.

[0069] Furthermore, server 104 can implement dynamic shard management, continuously monitoring the data volume growth and query load of each shard. Server 104 can periodically collect statistical information of each shard, including the number of data records in the shard, the storage space occupied, the frequency of query requests, and the average latency of query responses. When server 104 detects that the data volume of a shard exceeds the preset capacity limit, it can automatically trigger a shard expansion operation, alleviating the storage pressure on a single shard through data migration and redistribution. During the shard expansion process, server 104 can pause new data write operations to the target shard to ensure consistency and integrity during the data migration process. Server 104 analyzes the distribution characteristics of the data in the shards and, based on the time range of the timestamps or the hash value of the data records, reasonably divides the existing data into multiple data subsets. Server 104 can create new extended shards, configure the same index structure and field mapping as the original shards, and then migrate the data subsets to the corresponding extended shards. After data migration is complete, server 104 can update the index mapping relationship of the shards, modify the data routing rules, ensure that new data write requests can be correctly distributed to the corresponding shards, and finally restore normal data write operations. Conversely, when server 104 finds that the data volume of adjacent shards is lower than the preset lower threshold, it can perform a shard merging operation to optimize the utilization efficiency of storage resources. By integrating the data of multiple small shards into a single shard, the system's management overhead and query complexity are reduced.

[0070] Step S206: Extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0071] Data characteristics refer to statistical information extracted from monitoring data that reflects the essential attributes of the data, such as the data's continuity, coefficient of variation, trend indicators, and distribution patterns. Data monitoring charts refer to visual displays generated based on data characteristics, and can include various chart types such as line charts and area charts for displaying time series changes, bar charts for numerical comparisons, scatter plots for correlation analysis, and pie charts for displaying proportional relationships.

[0072] For example, server 104 can analyze the basic statistical characteristics of monitoring data, calculating the mean, variance, standard deviation, skewness, and kurtosis of each monitoring indicator. Furthermore, server 104 can detect the time-series characteristics of the data, identifying periodic patterns and trends through autocorrelation analysis and spectral analysis, such as periodic fluctuations in CPU utilization during working hours or long-term growth trends in memory usage. Regarding data continuity analysis, server 104 can assess the sampling density and time interval distribution of monitoring data to determine whether the data is suitable for time-series analysis or requires discrete data processing methods. Server 104 can also calculate the coefficient of variation and stability indicators of the data, identifying the degree of fluctuation and abnormal change patterns of monitoring indicators. Through multi-dimensional feature extraction analysis, server 104 can comprehensively grasp the inherent laws and distribution characteristics of monitoring data.

[0073] Based on the extracted data feature information, server 104 can automatically determine the most suitable visualization method according to the data's characteristic attributes. For example, when server 104 detects that the monitoring data has obvious time-series characteristics and the data changes continuously, it can prioritize line charts or area charts for display, clearly showing the data's changing trends and fluctuation patterns over time. For monitoring data that needs to compare the magnitude of values ​​in different categories or time periods, server 104 can choose bar charts for visualization, intuitively reflecting numerical differences through the comparison of bar heights. When monitoring data needs to show the relationship between the whole and its parts, server 104 can use pie charts to show the proportion of each component in the whole, such as the memory usage ratio of different applications or the space usage of each disk partition. For data that needs to analyze the correlation between two or more variables, server 104 can choose scatter plots for display, revealing the correlation and dependency between variables through the distribution pattern of data points. During the chart generation process, server 104 can also implement data smoothing techniques, using moving average algorithms to eliminate short-term noise and random fluctuations, enabling the charts to more clearly display the main trends and patterns of the data.

[0074] After generating the charts, server 104 can transmit the generated monitoring charts to the corresponding client devices. Server 104 can establish a client permission management system, determining the scope of monitoring data and chart types that users can access based on their roles and permission levels, ensuring the security of sensitive monitoring information and the effectiveness of access control. Server 104 can select appropriate distribution strategies based on the importance and timeliness requirements of the chart content. For performance indicator charts that require real-time monitoring, a push mode is used to proactively send chart updates to clients, achieving low-latency real-time communication through WebSocket connections. For charts of periodically reported types, server 104 can use a scheduled distribution mode, automatically generating and sending charts to designated clients according to a preset time plan. When a client actively requests a specific monitoring chart, server 104 can use an on-demand response mode, querying relevant data in real time and generating the corresponding chart to return to the client.

[0075] In some embodiments, server 104 can employ a web-based chart display and distribution scheme, creating a multi-level dashboard architecture within Kibana (a data visualization platform). This includes an overview dashboard to display the overall health status of all servers, detailed dashboards for in-depth monitoring and analysis of individual servers, and comparison dashboards for performance comparisons across multiple servers. Server 104 can be configured with various visualization chart components. Area charts can be used to display monitoring metrics such as CPU utilization and memory usage over time; bar charts can be used to compare monitoring data at different points in time, such as disk I / O read / write operations; line charts can be used to display trends in monitoring data, such as network traffic changes; and pie charts can be used to show the proportion of each part in the whole, such as disk space allocation.

[0076] For automated reporting, Server 104 can integrate with JasperReports Server, offering standardized report templates that include report titles, monitoring data time ranges, static charts and images exported from Kibana, data analysis summaries, and optimization recommendations. Server 104 can be configured with a scheduled task mechanism to automatically execute the report generation process at a set frequency and send the generated PDF reports to designated administrators and teams via email or other distribution methods.

[0077] In the above-mentioned monitoring data processing method, real-time system monitoring data of the target server is acquired, and the real-time system monitoring data is constructed into a distributed sequence index structure to obtain structured data. Based on the data volume and timestamp of each data item in the structured data, the structured data is fragmented to obtain target analysis data. This realizes intelligent partition management of data, which not only considers the balance of data storage capacity but also ensures the logical continuity of time series data, thereby optimizing the performance of distributed storage and the efficiency of data access. Data features in the target analysis data are extracted, and data monitoring charts are generated based on the data features. This realizes intelligent automatic selection of chart types and distributes the data monitoring charts to the corresponding clients, realizing automated transmission and distribution of monitoring results and improving the efficiency of monitoring data processing.

[0078] In one exemplary embodiment, such as Figure 3 As shown, step S206 includes steps S302 to S306. Wherein:

[0079] Step S302: Based on the data characteristics, determine the data type, numerical distribution, and time series data of the target analysis data, and select the chart type according to the data type, numerical distribution, and time series data.

[0080] The chart types include at least one of line charts, bar charts, area charts, scatter plots, and pie charts.

[0081] For example, in determining the data type, server 104 can detect the basic attributes of the monitoring data to identify whether the data is continuous numerical data or discrete categorical data. Continuous numerical data may include indicators that can continuously change within a certain numerical range, such as CPU utilization percentage, memory usage in bytes, and network transmission rate. Discrete categorical data may include indicators with clear categorical attributes, such as server status indicators, alarm level classifications, and network connection types. Server 104 can further analyze the measurement scale characteristics of the data, distinguishing between interval data and ratio data. Interval data has equal intervals but no absolute zero, while ratio data has not only equal intervals but also a meaningful zero. In data accuracy analysis, server 104 can evaluate the significant digits and measurement accuracy of the monitoring data, determine the accuracy requirements and rounding rules for data display, and ensure that the numerical displays in charts are both accurate and easy to read.

[0082] In terms of numerical distribution characteristic analysis, server 104 can calculate statistical descriptive quantities of monitored data, including central tendency indicators such as arithmetic mean, median, and mode, as well as dispersion indicators such as variance, standard deviation, and interquartile range. Server 104 can identify the distribution pattern characteristics of data by constructing frequency distribution tables and histograms, determining whether the data conforms to a normal, skewed, uniform, or other special distribution patterns. Server 104 can also calculate the skewness and kurtosis coefficients of the data. Skewness reflects the symmetry of the data distribution; positive skewness indicates a longer right tail, and negative skewness indicates a longer left tail. Kurtosis reflects the kurtosis of the data distribution; high kurtosis indicates that the data is more concentrated near the mean, and low kurtosis indicates a relatively flat distribution. Through outlier detection algorithms, server 104 can identify outliers and extreme values ​​in the data, using box plots or Z-scores to mark data points that exceed the normal range.

[0083] In time series feature analysis, Server 104 can detect whether the monitored data has obvious time dependence and serial correlation. Server 104 can calculate the autocorrelation function of the data and analyze the degree of correlation between data at different time lags. A high autocorrelation coefficient indicates that the data has strong time series characteristics and is suitable for time series visualization methods. Server 104 can identify the long-term development trend of the data through trend analysis algorithms, including upward, downward, or stationary trends. The strength and direction of the trend will affect the display of charts and the design of predictive functions. In terms of seasonality and periodicity analysis, Server 104 can use frequency domain analysis methods to identify periodic patterns in the data. For example, CPU utilization may exhibit daily periodic fluctuations.

[0084] Based on the comprehensive analysis of the above data types, numerical distributions, and time series characteristics, when the monitored data is continuous numerical data with obvious time series characteristics, server 104 can prioritize line charts for display. For data requiring comparison of numerical values ​​across different categories or time periods, server 104 can choose bar charts. When the data has both time series characteristics and needs to demonstrate cumulative effects, server 104 can choose area charts. For data requiring analysis of the correlation between two or more variables, server 104 can choose scatter plots for visualization. When the monitored data needs to demonstrate the proportional relationship between the whole and its parts, server 104 can use pie charts for visualization.

[0085] Step S304: Generate the corresponding chart configuration parameters according to the selected chart type to obtain the data monitoring chart.

[0086] For example, in configuring the line chart, server 104 can first determine the axis settings. The horizontal axis corresponds to the time dimension, automatically selecting an appropriate time scale interval based on the data's time span and sampling frequency. For instance, hourly scales are used for hourly data, and date scales are used for daily data, ensuring a balance between readability and information density. The vertical axis corresponds to the numerical range of the monitoring indicators. Server 104 can automatically determine the starting and ending values ​​of the axis based on the data's maximum, minimum, and distribution characteristics, using reasonable scale intervals to ensure accurate numerical display. Server 104 can configure the visual style parameters of the line, including line color, thickness, and line type. Different color schemes are assigned based on the importance and category of the monitoring indicators; performance indicators use striking colors, while ordinary indicators use soft colors, ensuring the chart's visual hierarchy and recognizability. Regarding data point marking, server 104 can set the shape, size, and fill style of the marker points. Abnormal data points are highlighted with special markers, facilitating quick identification of performance anomalies and problem areas by users. Furthermore, in the bar chart configuration, server 104 can calculate the optimal bar width based on the number of data categories and the size of the chart display area, ensuring appropriate spacing between bars. For area chart configuration, server 104 can add area fill settings to the existing line chart configuration. In the scatter plot configuration, server 104 can set the visual attributes and distribution characteristics of the data points.

[0087] Furthermore, server 104 can select a data report template based on the chart type and data type corresponding to the data monitoring chart; configure the data source index according to the data report template; generate a data report based on the data report template, data source index, and data monitoring chart; determine distribution permissions based on the data type; and distribute the data report to the corresponding client according to the distribution permissions.

[0088] For example, server 104 can utilize Kibana's visualization component library to configure corresponding visualization controls for different types of monitoring data, including time series visualization controls for time series data, metric visualization controls for aggregated data, and map visualization controls for geographic data. Server 104 can configure dashboard layouts in Kibana, organizing multiple related charts into a single dashboard to achieve comprehensive display and correlation analysis of monitoring data. Server 104 can also set up an automatic refresh mechanism, configuring the chart refresh interval based on the update frequency of the monitoring data to ensure that the chart content reflects the latest system status in real time.

[0089] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0090] Based on the same inventive concept, this application also provides a monitoring data processing apparatus for implementing the monitoring data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more monitoring data processing apparatus embodiments provided below can be found in the limitations of the monitoring data processing method described above, and will not be repeated here.

[0091] In one exemplary embodiment, such as Figure 4 As shown, a monitoring data processing device is provided, including: a data acquisition module 402, a data processing module 404, and a data distribution module 406, wherein:

[0092] The data acquisition module 402 is used to acquire real-time system monitoring data of the target server;

[0093] The data processing module 404 is used to construct a distributed sequence index structure from real-time system monitoring data to obtain structured data, and to perform data sharding on the structured data according to the data volume and timestamp of each data item in the structured data to obtain target analysis data.

[0094] The data distribution module 406 is used to extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

[0095] In one embodiment, the data distribution module 406 is specifically used to determine the data type, numerical distribution, and time series data of the target analysis data according to the data characteristics, and select a chart type according to the data type, numerical distribution, and time series data; the chart type includes at least one of line chart, bar chart, area chart, scatter plot, and pie chart; and generate corresponding chart configuration parameters according to the selected chart type to obtain a data monitoring chart.

[0096] In one embodiment, the data distribution module 406 is further configured to select a data report template according to the chart type and data type corresponding to the data monitoring chart; configure the data source index according to the data report template; generate a data report according to the data report template, the data source index and the data monitoring chart; determine the distribution permission according to the data type; and distribute the data report to the corresponding client according to the distribution permission.

[0097] In one embodiment, the data processing module 404 is specifically used to calculate the data query frequency and data growth rate of each data item based on the timestamp and data volume; calculate the data capacity of a single shard based on the data volume and data growth rate; and shard the structured data according to at least one of the following dimensions: time dimension, server level dimension, or data type dimension, based on the data query frequency and data capacity, to obtain the target analysis data.

[0098] In one embodiment, the apparatus further includes: a data monitoring module, configured to acquire data volume growth monitoring data and load monitoring data for each segment in the target analysis data; when the data volume of a segment exceeds a preset upper limit, allocate the data of the segment to at least one extended segment; and when the data volume of adjacent segments of a segment is lower than a preset lower limit, merge the segment and adjacent segments.

[0099] In one embodiment, the data monitoring module is further configured to pause data write operations on the shard, and divide the data in the shard into multiple data subsets based on the timestamp and / or corresponding hash value of the data in the shard; create new extended shards and migrate the data subsets to the corresponding extended shards; update the index mapping relationship between the shards and the extended shards, and resume data write operations.

[0100] Each module in the aforementioned monitoring data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0101] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores real-time system monitoring data and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a monitoring data processing method.

[0102] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0104] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0105] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0106] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0109] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing monitoring data, characterized in that, The method includes: Obtain real-time system monitoring data from the target server; The real-time system monitoring data is constructed into a distributed sequence index structure to obtain structured data. Based on the data volume and timestamp of each data item in the structured data, the structured data is fragmented to obtain target analysis data. Extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

2. The method according to claim 1, characterized in that, The step of generating data monitoring charts based on the data features includes: Based on the data characteristics, the data type, numerical distribution, and time series data of the target analysis data are determined, and the chart type is selected according to the data type, the numerical distribution, and the time series data; the chart type includes at least one of line chart, bar chart, area chart, scatter plot, and pie chart; Based on the selected chart type, generate the corresponding chart configuration parameters to obtain the data monitoring chart.

3. The method according to claim 2, characterized in that, The step of distributing the data monitoring charts to the corresponding clients includes: Select a data report template based on the chart type and data type corresponding to the data monitoring chart; Configure the data source index according to the data report template, and generate a data report based on the data report template, the data source index, and the data monitoring chart; Based on the data type, the distribution permission is determined, and based on the distribution permission, the data report is distributed to the corresponding client.

4. The method according to any one of claims 1 to 3, characterized in that, The step of segmenting the structured data according to the data volume and timestamp of each data item in the structured data to obtain the target analysis data includes: Based on the timestamp and the amount of data, calculate the data query frequency and data growth rate of each data item; Calculate the data capacity of a single shard based on the data volume and the data growth rate; Based on the data query frequency and the data volume, the structured data is sharded according to at least one of the following dimensions: time dimension, server level dimension, or data type dimension, to obtain the target analysis data.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the data volume growth monitoring data and load monitoring data of each segment in the target analysis data; If the data volume of the fragment exceeds a preset upper limit, the data of the fragment will be allocated to at least one extended fragment; If the data volume of adjacent fragments of the fragment is lower than a preset lower limit, the fragment and the adjacent fragments are merged.

6. The method according to claim 5, characterized in that, The step of allocating the data of the fragment to at least one extended fragment includes: Pause the data writing operation on the shard, and divide the data in the shard into multiple data subsets according to the timestamp and / or the corresponding hash value of the data in the shard; Create a new extended shard and migrate the data subset to the corresponding extended shard; Update the index mapping relationship between the shard and the extended shard, and resume data write operations.

7. A monitoring data processing device, characterized in that, The device includes: The data acquisition module is used to acquire real-time system monitoring data from the target server. The data processing module is used to construct the real-time system monitoring data into a distributed sequence index structure to obtain structured data, and to perform data fragmentation on the structured data according to the data volume and timestamp of each data item in the structured data to obtain target analysis data. The data distribution module is used to extract data features from the target analysis data, generate data monitoring charts based on the data features, and distribute the data monitoring charts to the corresponding clients.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images