An Optimization Method for Mass Data Plotting Based on the Maximum Triangle Three-Section Algorithm

Abstract

The present invention discloses a method for optimizing the drawing of massive data based on the maximum triangle three-segment algorithm. The present invention first downsamples the data based on the maximum triangle three-segment algorithm, attempts to compress the original data and maintain the detailed features as much as possible. And for locally steep segmented data, an improvement of dynamic adjustment of the segmentation is made. Then, the data whose magnitude exceeds the threshold is sliced, and for each sliced data, ECharts will create an instance separately for drawing to obtain a chart of the sub-sliced data. Finally, for the obtained sub-charts, the cascading style sheet language is used to locate them to the same position, and they are superimposed to form a unique and complete graph. By adopting the present invention, the data points used for drawing can be greatly reduced without losing the details of the original data as much as possible, thereby visualizing the industrial massive data with high efficiency and high performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of data visualization, and in particular relates to a mass data drawing optimization method based on a maximum triangle three-segment algorithm. Background Art

[0002] In recent years, remote operation and maintenance (O&M) and online monitoring systems have become widely used in industry. In industrial monitoring scenarios, the visualization of time series data is an unavoidable topic. Many monitoring systems require frequent collection and reporting of data from field devices, displaying it in real-time visualization charts. This data is often updated frequently, and each update results in a large amount of time series data. This results in a very large dataset.

[0003] This massive amount of data presents several challenges when used: First, features are unclear. The data volume is large and fluctuates frequently, and the data points overlap when connected by broken lines. This makes it difficult to identify significant trends, whether for subsequent data mining or direct plotting. This hinders data analysis and visualization. Second, the excessive number of data points poses challenges to graphical display performance. The rendering performance of system page visualization technologies such as ECharts and HighCharts degrades rapidly as the number of plot points increases. Updates and redraws can be slow, and memory usage is high, severely impacting system performance.

[0004] Therefore, in industrial monitoring scenarios, it is necessary to optimize the processing of massive amounts of time series data. To address the issue of large amounts of time series data, downsampling can be performed while preserving the original data's fluctuation details, trends, and shapes as much as possible. Specifically, quantitative analysis involves analyzing the statistical characteristics of the series (such as mean, extreme value, and variance) and morphological features (such as distance, angle, and slope).

[0005] The LTTB algorithm proposed by Sveinn of the University of Iceland is currently the most suitable for industrial use. However, in practice, this algorithm is prone to missing "spikes" in steep data points and is imperfect. Chen Jiandong and others at Guangzhou Huya Company used a method called mean sampling (such as PAA) before LTTB processing to reduce spikes before performing LTTB. However, PAA tends to lose more details of the data curve shape, which is not worth the effort. Wang Lifeng and others first used the time difference between the current and previous data points as a weight, selected a certain weight sum as a threshold, and segmented the data for LTTB. While the segmentation concept is correct, using time difference as the weight ignores the fact that spikes are caused by sudden changes in values, not by excessive time differences. Moreover, the threshold selection is not rigorous, with no suitable fixed value, making it difficult to select an appropriate threshold in practice. Summary of the Invention

[0006] In view of the above problems, the present invention proposes an optimization method for drawing massive data based on the maximum triangle three-segment algorithm, aiming to propose an improved method for LTTB downsampling of massive time-series data applicable to industrial scenarios, and a solution to make the subsequent drawing process perform better.

[0007] The present invention includes the following steps:

[0008] Step S1: Obtain the collected data and perform downsampling based on the maximum triangle three-segment algorithm (Large Triangle ThreeBuckets, LTTB) and its dynamic improvement algorithm;

[0009] Step S2: Set a threshold to slice the over-large data;

[0010] Step S3: Then create an ECharts instance and render and draw each given slice of data separately. Finally, stack each sub-chart to form a complete chart.

[0011] Furthermore, in step S1, it is improved based on the maximum triangle three-segment algorithm (Large Triangle ThreeBuckets, LTTB). For most regular graphs, the present invention uses the LTTB algorithm for downsampling, but for some special shapes and severely uneven data distributions, it is improved to use a dynamic segmentation algorithm, that is: when the data is relatively flat, more points are regarded as a segment, when the data is steep, fewer points are regarded as a segment, and when selecting the effective triangle area, by adopting this method, a good sampling effect can be achieved for special data curves with irregular change trends.

[0012] Furthermore, in step S2, for data still at a relatively large magnitude, a reasonable threshold is set to slice the data, thereby further reducing the data magnitude.

[0013] Furthermore, in step S3, an instance is created using data visualization technology, and the data is rendered into a visual chart. For the sliced data, multiple drawing instances need to be created, and when finally forming the chart, multiple instances are stacked to form a complete chart.

[0014] Furthermore, the data visualization technology described is the open-source product ECharts. ECharts is an open-source JavaScript visualization chart library of Baidu, Inc., with rich chart types and a self-developed efficient rendering engine ZRender, which can achieve elegant and friendly responsive chart designs.

[0015] Furthermore, the ECharts chart selected is a line chart. Since the visualization of time series data is more suitable to be represented by lines, the present invention takes the line as an example. The ECharts line chart should consider the superposition problem of the sliced data mentioned in steps S3 and S4. Specifically, the drawing of the ECharts chart should create an instance using the given data and hand it over to the rendering engine ZRender for drawing. If data slicing is performed, separate instances should be created for each sliced data, and the multiple sub-charts obtained by rendering should be positioned separately and located at the same position to achieve the purpose of superposing them into a complete chart.

[0016] Furthermore, the Cascading Style Sheets (CSS) language is used to complete the positioning and superposition of the sub-charts. The sub-charts are set to have the same size (width, height), position (top, left), and positioning method (absolute), the parent container is set to relative positioning, and each sub-chart is absolutely positioned to the same position within the container to be superposed into a complete chart.

[0017] The present invention mainly proposes the following improvements:

[0018] First, to address the problem of excessive data points, a downsampling method based on the Large Triangle Three Buckets (LTTB) algorithm and its dynamic improvement algorithm based on distance and dispersion is proposed. The Piecewise Aggregate Approximation (PAA) algorithm mentioned above maintains a sliding window and takes the average of the data within a time period in each window. However, for industrial time series data, sequences that change frequently after mean processing will seriously lose shape details, and it is difficult to maintain shape features, so it is only suitable for data with stable amplitude changes.

[0019] In contrast to the PAA which calculates the mean to sample points, the present invention is based on LTTB. LTTB selects sampling points by calculating the maximum effective area. The effective area of a point is defined as the area of the triangle formed by the two adjacent points. By selecting the point with the largest effective area within the current segment, the distance, angle, and contour features between points can be effectively maintained. Although the algorithm is relatively complex, it is very suitable for processing massive and frequently changing time series data. <SHAPE>

[0020] In addition, to handle extreme cases where the local area is too steep, the present invention improves the method of dynamically determining the segmentation size based on distance and dispersion. The main principle of the method is to fit an asymptote, and the distance from the data points to the asymptote is used to characterize their dispersion. The sum of all the distances is trisected. Thus, when encountering a spike, as long as the fitting line is as accurate as possible, there is a high probability that the spike will fall into a single equal segment and thus be adopted. Because other points approach the fitting line and will not or only produce very small distances.

[0021] Second, to address the problem that the data may still be too large after downsampling, a threshold is set to further slice the data, and each slice of data is rendered and drawn separately. At the same time, in view of the characteristic that the time consumption increases sharply when ECharts draws large data, it is proposed to divide the overall data, then render them separately, and finally stack and align the sub-charts into a complete chart to complete the drawing. In this way, the data volume can be controlled within an acceptable range.

[0022] Advantages of the present invention: The present invention can significantly reduce the data points used for drawing with minimal loss of the details of the original data, control the data volume within a reasonable threshold, and directly hand the binary data to the drawing program for drawing, so as to efficiently and with high performance visualize and draw the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is the overall flowchart of the present invention.

[0024] Figure 2 is a schematic diagram of the improved downsampling algorithm used in the present invention.

[0025] Figure 3 is a schematic diagram of the process of drawing after data slicing in the present invention.

[0026] Figure 4 is the test drawing effect diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] [[ID=]29]To make the technical solution of the present invention clearer and more understandable, the following will be described in detail with reference to the accompanying drawings. As shown in the present invention Figure 1 shown, the specific steps are as follows:

[0028] Step S1, obtain the collected data and perform downsampling based on the Large Triangle ThreeBuckets (LTTB) algorithm and its dynamic improvement algorithm. The steps of the LTTB algorithm are as follows:

[0029] S1-1. Determine the segmentation size threshold: To facilitate changing the segmentation size, the segmentation size is passed as a parameter (threshold) to the algorithm. In this way, if 100 times sampling is required, only the parameter threshold = (total data size / multiple) needs to be passed. The total number of data points is evenly divided into all these segments, and a total of threshold segments are formed. Additionally, to ensure that the first and last points can be selected after the data is segmented, the first and last points each occupy a separate segment.

[0030] S1-2. Select the first point (i.e., the first segment).

[0031] S1-3. Starting from the second segment, traverse all points in the segment, calculate the effective triangular region of each point, and select the point with the largest effective region area as the selected point (sampling point) of this segment. Among them, the so-called effective triangular region refers to the triangular region with three points [the selected point A of the previous segment, the current point F, and the average point B of the next segment] as vertices.

[0032] S-4. Traverse until the last point (i.e., the last segment) is selected, and the algorithm ends.

[0033] The problem with the LTTB algorithm is that it evenly divides all segments, which performs poorly in some cases where the data curve is relatively steep. Because fewer points can reflect details in the flat areas, while more points are needed to reflect details in the steep areas. In order to save the time consumed by segmenting, LTTB simply divides evenly. Therefore, the present invention makes an improvement in dynamic segmentation here, that is, an improvement to step (S1-1) of the above algorithm. Thus, step (S1-1) is improved to the following algorithm steps:

[0034] (1-1) Still pass the segmentation size as a parameter (threshold) to the algorithm. First, divide the segments evenly as in LTTB, and the first and last points each occupy a separate segment.

[0035] (1-2) Starting from the second segment, traverse all points in the segment (a total of m = total / threshold). Taking the selected point A of the previous segment and the average point B of the next segment as the asymptote L, calculate the perpendicular distance from each point F' in the segment to the line L to obtain an array SS and the sum SSE of all perpendicular distances.

[0036] (1-3)With target = 1 / 3*SSE as the target value, the array SS is trisected. For the found trisection indices, they are also used as segmentation points for segmentation. Since the vertical distance to a certain extent characterizes the degree of dispersion relative to the asymptote, the segmentation of one SSE is divided into three relatively uniform segments of target, which can relatively dynamically cope with the severe situation of rapid changes. Among them, if there is no trisection point exactly equal to the target, the points that are relatively close on the left and right are taken.

[0037] (1-4)After such segmentation, the above-mentioned step 2 of the LTTB algorithm can be carried out. The final improved algorithm process is as Figure 2 shown.

[0038] Step S2, data slicing. Set a threshold. For the data that is still large after downsampling, perform slicing operations and then enter the next step. Experiments show that the number of simultaneously drawn instances should not be too many, and the drawing consumption is almost the same when the data volume is below one hundred. Therefore, the present invention sets: the maximum threshold for the number of sliced data pieces is 10, and the minimum threshold for the capacity of a single data piece is 200.

[0039] Step S3, drawing and rendering. Taking ECharts drawing as an example in the present invention, the most time-consuming operation is drawing data points. However, due to the implementation of the downsampling algorithm step in the present invention, when considering the actual effect, the algorithm time consumption for processing data also needs to be considered. This drawing process mainly includes:

[0040] S3-1, create an ECharts instance: separately render and draw each slice of data given.

[0041] S3-2, use Cascading Style Sheets language to position and stack each sub-chart to form a complete chart. Specifically, only need to set the parent container as relative positioning, position each sub-chart absolutely to the same position within the container, and then only keep one copy of the axis scale of the chart, and then it can be stacked. Although there are multiple sub-charts, this chart is complete in the visual sense of the human eye. The specific process is shown in Figure 3 .

[0042] The technical solution is as described in the above steps. Next, tests are carried out and the results are summarized to illustrate the optimization results that can be achieved by the present invention. The test steps are as follows:

[0043] A. Generate test data. Randomly generate 100,000 time series data. The specific generation steps are as follows:

[0044] (1) Generate the past timestamp base: base = new Date(1988,9,3)

[0045] (2) Traversal, with the time series cumulative base base each time: base += 24 * 3600 * 1000, now = new Date(base)

[0046] (3) Push the data set into [now, randomly selected value]

[0047] (4) After traversing 100,000 times, 100,000 randomly generated values with continuous time series are obtained.

[0048] B. Downsampling. Apply different sampling ratios of 1, 10, 100, 500, 1000, and 10000 to the 100,000 data points, and use the algorithm in step S1 for downsampling. The degree of detail retention in the final drawing is as Figure 4 , and the test data is shown in Table 2 below.

[0049] C. Data segmentation and rendering. For the data samples with sampling ratios of 1, 10, and 100, the sampled data still exceeds the threshold. Use the methods in steps S2 and S3 to segment and separately render the instances of the data. The test data is shown in Table 3 below.

[0050] The final test data is as follows:

[0051] I. Data index settings

[0052] Test data 100000 Downsampling ratio 1、10、100、500、1000、10000 Data segmentation threshold 10 (maximum data shards), 200 (single minimum capacity)

[0053] II. Drawing test data

[0054]

[0055] III. Drawing test data after data segmentation

[0056]

[0057] As shown by the above test data, by using the drawing optimization method of downsampling and data segmentation in the present invention, for each data magnitude reduction in downsampling the data, the drawing time consumption can be reduced by approximately 70 - 80%, and the memory occupancy can be reduced by approximately 20 - 30% at the same time, with good effects. Especially when appropriate sampling ratios and thresholds are selected, the effects are very good. For example, combined with Figure 4 the drawing effect diagram and tabular data, it can be seen that when downsampled to 1000 data points and the threshold is set to 10 * 200 as described in the scheme, the detailed features of the graph are well retained, and the peak and trough points are retained even when downsampled to 10 points. At the same time, the measured drawing time consumption and memory occupancy data are also relatively ideal. In summary, it can be shown that the present invention has achieved the expected effects.

Claims

1. A method for optimizing the drawing of massive data based on the maximum triangle three-segment algorithm, characterized in that It includes the following steps: Step S1: Obtain the collected data and perform downsampling based on the improved maximum triangle three-segment algorithm; Step S2: Set the maximum threshold for the number of sliced data pieces and slice the over-large data; Step S3: Create an ECharts instance, render and draw each given sliced data separately; Overlay each sub-chart to form a complete chart; In the improved maximum triangle three-segment algorithm described in Step S1, when determining the segmentation size: Treat the segmentation size as a parameter and pass it to the maximum triangle three-segment algorithm. The first and last points each occupy a separate segment; Starting from the second segment, traverse all points in the segment. Using the selected point A of the previous segment and the average point B of the next segment as the asymptote L, calculate the perpendicular distance from each point F' in the segment to the line L to obtain the array SS and the sum of all perpendicular distances SSE; Taking target = 1 / 3 * SSE as the target value, trisect the values of the array SS. For the found trisect indices, also use them as segmentation points for segmentation; In Step S3, use data technology to create an ECharts instance, render the data into a visual chart; For the sliced data, multiple drawing instances need to be created, and when finally forming the chart, multiple instances are overlaid to form a complete chart; Use cascading style sheet language to complete the positioning and overlay of sub-charts: Set the sub-charts to the same size, position, and positioning method. Set the parent container to relative positioning, and absolutely position each sub-chart to the same position within the container, and overlay them to form a complete chart.

Images