A data visualization monitoring method and device and a storage medium

Abstract

The application relates to a data visualization monitoring method and device and a storage medium, the method comprising: collecting Grafana interface data in a timing round-robin collection mode, filtering the collected interface data to generate filtered interface data, filling the filtered interface data into a database chart, simultaneously acquiring a monitoring configuration of a Web page, monitoring the database chart according to the monitoring configuration, judging whether the filtered interface data in the database chart is abnormal, modifying the state of the database chart according to the judgment result, and returning the state of the database chart and the corresponding filtered interface data to the Web page. Thus, the application ensures the timeliness of data in the database chart in the mode of collecting interface data in a timing round-robin mode, filters the interface data, improves the data processing capacity and the data monitoring efficiency, and supports the user to directly perform monitoring configuration on the Web page, so that the user experience is simple, convenient and optimized.

Description

Technical Field

[0001] This invention relates to the field of data monitoring technology, and in particular to a data visualization monitoring method, device, and storage medium. Background Technology

[0002] Current visualization monitoring methods have evolved from the initial alarm configuration approach, which resulted in a massive number of alarms due to the large number of monitored objects and alarm metrics, making accurate fault detection impossible. Now, some open-source visualization components such as Kibana, Superset, and Grafana are used to achieve data visualization monitoring. However, with the explosion of data volume, the number of monitored objects, monitoring metrics, and monitoring dimensions are also exploding. Relying solely on manual methods is no longer sufficient to handle daily data monitoring, affecting work efficiency. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a data visualization monitoring method, device and storage medium to improve the efficiency of data visualization monitoring.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a data visualization and monitoring method, comprising:

[0006] The Grafana interface data is collected periodically and in rotation, filtered, and then the filtered interface data is populated into the database chart.

[0007] Obtain the monitoring configuration of the web page, retrieve the corresponding database chart based on the chart information in the monitoring configuration, and determine whether there are any anomalies in the filtered interface data of the database chart according to the alarm rules in the monitoring configuration. If so, mark the status of the database chart as abnormal; otherwise, mark it as normal, and return the status of the database chart and the corresponding filtered interface data to the web page.

[0008] The beneficial effects of this invention are as follows: Since Grafana's interface data is complex and variable, filtering the obtained Grafana interface data before filling it into the database chart improves the data processing capability. At the same time, the method of collecting data in a timed and rotating manner ensures the timeliness of the interface data in the database chart. Furthermore, the monitoring of the filtered interface data in the database chart can be achieved by directly configuring the monitoring on the web page, which improves the efficiency of data visualization and monitoring.

[0009] Optionally, the step of periodically collecting Grafana's interface data, filtering the interface data, and generating filtered interface data includes:

[0010] The Grafana API data is collected periodically and in rotation. Based on the type of the API data, filtering rules are applied to filter the API data, and filtering results are generated.

[0011] Determine whether the filtering result conforms to the data retrieval rules. If it does not conform, apply the filtering rules according to the type of the filtering result to perform filtering processing until the filtering result conforms to the data retrieval rules. Otherwise, generate filtered interface data based on the filtering result.

[0012] As described above, filtering rules are invoked based on the type of interface data to filter the interface data, improving the flexibility of data processing. The judgment of data retrieval rules is performed on the filtering results after the interface data has been filtered, which reduces the amount of computation for data judgment and improves the efficiency of data processing. At the same time, for filtering results that do not meet the data retrieval rules, filtering rules will be invoked based on the type of the filtering result to filter them until the data retrieval rules are met, thereby improving the accuracy of the data.

[0013] Optionally, the step of applying filtering rules to filter the interface data based on the type of the interface data includes:

[0014] Determine whether the interface data has multiple layers of nested data. If so, start from the innermost layer of the interface data and proceed outwards to determine the type of each layer of data. Then, recursively call the corresponding filtering rules for each layer of data based on its type to perform filtering processing. Otherwise, directly call the corresponding filtering rules for filtering processing based on the type of the interface data.

[0015] As described above, it can handle complex interface data with multiple layers of nested data, has strong data processing capabilities, and recursively calls the corresponding filtering rules according to the type of each layer of data, which not only improves the flexibility of data processing but also improves the accuracy of data.

[0016] Optionally, the type of the interface data includes arrays and objects. When it is an array, array filtering rules are called for filtering; when it is an object, object filtering rules are called for filtering.

[0017] As described above, the types of interface data are divided into arrays and objects. Different filtering rules are invoked for different types of interface data to improve the flexibility of data processing.

[0018] Optionally, marking the state of the database chart as abnormal further includes:

[0019] The database charts are color-coded.

[0020] As described above, database charts in abnormal states will be color-coded to prominently alert relevant personnel to abnormal database charts, thereby improving the timeliness of anomaly detection and increasing work efficiency.

[0021] Optionally, the monitoring configuration chart information includes page name, chart name, and chart weight, and returning the status of the database chart and the corresponding filtered interface data to the web page includes:

[0022] The database charts are categorized according to the page names to generate categorized database charts;

[0023] The categorized database charts are sorted according to their status. Database charts with an abnormal status are placed at the top. When multiple categorized database charts have an abnormal status, they are sorted according to their chart weights. When the chart weights are the same, they are sorted according to their chart names.

[0024] When multiple categorized database charts are in a normal state, the categorized database charts are sorted according to the chart weight. When the chart weights are the same, the categorized database charts are sorted according to the chart name to generate a sorted database chart.

[0025] The status of the sorted database chart and the corresponding filtered interface data are returned to the web page.

[0026] As described above, categorizing database charts by page name facilitates the subsequent convergence of database charts from the same scenario or system into a single panel for display, improving the effectiveness of web page visualization monitoring. Furthermore, categorized data charts are sorted based on their status, weight, and name, enhancing the flexibility of database chart display on the web page and enabling relevant personnel to promptly identify and view abnormal database charts, thus optimizing the user experience.

[0027] In a second aspect, a data visualization and monitoring device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data visualization and monitoring method described in the first aspect.

[0028] Thirdly, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed, it implements the data visualization and monitoring device described in the first aspect.

[0029] The data visualization and monitoring device provided in the second aspect and the computer-readable storage medium provided in the third aspect have the same technical effects as described in the data visualization and monitoring method provided in the first aspect. Attached Figure Description

[0030] Figure 1 This is a flowchart of a data visualization monitoring method provided in an embodiment of the present invention;

[0031] Figure 2 This is a flowchart illustrating a data visualization and monitoring method provided in an embodiment of the present invention.

[0032] Figure 3 This is a diagram illustrating the monitoring configuration of a web page according to an embodiment of the present invention;

[0033] Figure 4 This is a display diagram showing the status of the database chart received by the web page and the corresponding filtered interface data in an embodiment of the present invention.

[0034] Figure 5 This invention provides a data visualization and monitoring device.

[0035] [Explanation of Labels in the Attached Image]

[0036] 1. A data visualization and monitoring device;

[0037] 2. Processor;

[0038] 3. Memory. Detailed Implementation

[0039] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0040] Example 1

[0041] Please refer to Figures 1 to 4 This invention provides a data visualization and monitoring method, comprising the following steps:

[0042] S1. Collect Grafana interface data in a regular cycle, filter the interface data, generate filtered interface data, and fill the filtered interface data into the database chart.

[0043] Grafana is a visualization tool.

[0044] like Figure 2 As shown, Grafana interface data is collected in a timed, rotating manner. The time is set to one minute, meaning that Grafana interface data is collected once every minute. The time setting can be adjusted according to specific circumstances. The collected interface data will be filtered to generate filtered interface data, which will then be filled into the database chart to update and populate the database chart data.

[0045] At this point, the process of periodically collecting Grafana interface data in step S1, filtering the interface data, and generating filtered interface data includes:

[0046] S11. Periodically collect Grafana interface data in rotation, apply filtering rules to filter the interface data according to the type of the interface data, and generate filtering results;

[0047] like Figure 2 As shown, for the collected Grafana interface data, filtering rules are invoked to filter the data according to its type, thereby generating the filtering results.

[0048] At this point, step S11, which involves calling filtering rules based on the type of the interface data, includes:

[0049] S111. Determine whether the interface data has multiple layers of nested data. If so, start from the innermost layer of the interface data and proceed outwards to determine the type of each layer of data. Then, recursively call the corresponding filtering rules for filtering based on the type of each layer of data. Otherwise, directly call the corresponding filtering rules for filtering based on the type of the interface data.

[0050] like Figure 2 As shown, when filtering interface data, it first determines whether there are multiple layers of nested data. When there are multiple layers of nested data, it starts from the innermost layer and proceeds outward to determine the type of each layer of data. Then, it recursively calls the corresponding filtering rules to perform filtering based on the type of each layer of data. For interface data without multiple layers of nested data, i.e., single-layer interface data, it directly calls the filtering rules based on the type of interface data to perform filtering.

[0051] At this point, step S11 includes:

[0052] S112. The types of interface data include arrays and objects. When it is an array, the array filtering rules are called for filtering. When it is an object, the object filtering rules are called for filtering.

[0053] like Figure 2 As shown, the interface data is divided into two types: array and object. For different types of interface data, filtering rules are applied according to their types. That is, when the interface data is an array, the array filtering rules are applied, and when the interface data is an object, the object filtering rules are applied. The array filtering rules and object filtering rules are set in advance by the operations and maintenance personnel according to the actual situation.

[0054] In one specific embodiment, the interface data is: [{"timestamp":"2023-01-13T04:00:00.000Z","result":[{"x":22.0},{"x":24.0}]},{"timestamp":"2023-01-13T04:01:00.000Z","result":[{"x":30.0},{"x":40.0}]}]. The current interface data type is an array. Based on the "result[max].x" rule pre-set by the operations and maintenance personnel, the system will automatically identify the type and call the corresponding filtering rule for filtering processing, that is, retaining the maximum value of x in the latest result array within the outermost array. If the interface data to be processed is: {"took":429,"responses":{"took":429,"timed_out":false,"}}, then the type of the current interface data is an object. Based on the object type, the filtering rule "response.took" set in advance by the operations and maintenance personnel is called to perform filtering processing, that is, to retain the value of toook.

[0055] S12. Determine whether the filtering result conforms to the data retrieval rules. If it does not conform, call the filtering rules according to the type of the filtering result to perform filtering processing until the filtering result conforms to the data retrieval rules. Otherwise, generate filtered interface data according to the filtering result.

[0056] like Figure 2As shown, the filtering results generated in step S11 will be verified again to determine whether the filtering results conform to the data retrieval rules. The data retrieval rules are set in advance by the operation and maintenance personnel, such as the data retrieval rules for the maximum value, the minimum value, the average value, the latest value, etc. The specific data retrieval rules can be set according to the actual situation. For the filtering results that do not conform to the data retrieval rules, the filtering rules will be called again according to the type of the filtering results for further filtering. This process is repeated until the filtering results that conform to the data retrieval rules are obtained, thereby generating the corresponding filtered interface data.

[0057] S2. Obtain the monitoring configuration of the Web page, obtain the corresponding database chart according to the chart information in the monitoring configuration, and determine whether there is any abnormality in the filtered interface data of the database chart according to the alarm rules in the monitoring configuration. If so, mark the status of the database chart as abnormal; otherwise, mark it as normal, and return the status of the database chart and the corresponding filtered interface data to the Web page.

[0058] like Figure 3 As shown, users can configure monitoring on the web page, including setting the chart information and alarm rules to be monitored. Based on the chart information and alarm rules in the monitoring configuration, the system can determine whether there are any anomalies in the filtered interface data in the database chart. For database charts with anomalies, their status will be changed to abnormal; otherwise, for database charts without anomalies, their status will be changed to normal. Finally, the status of the database chart and the corresponding filtered interface data will be returned to the web page, realizing visualized monitoring of the data.

[0059] In one specific embodiment, the chart name in the monitoring configuration of the web page is: "Comparison of Today's and Yesterday's Business Hall Channel Failure Count". That is, the database chart with the name "Comparison of Today's and Yesterday's Business Hall Channel Failure Count" needs to be monitored. The alarm rule is: if it is greater than the alarm threshold, an alarm will be triggered. The alarm threshold is set to 0, which means that the filtered interface data of the database chart is compared with the alarm threshold 0. If the filtered interface data is greater than the alarm threshold 0, an alarm will be triggered, which is an anomaly.

[0060] At this point, marking the state of the database chart as abnormal in step S2 also includes:

[0061] S21. Color-code the database charts.

[0062] In this embodiment, database charts in abnormal states will be highlighted with color. In this case, the color is set to red, that is, database charts in abnormal states will be marked in red. The color setting can be adjusted according to the actual situation.

[0063] At this point, step S2 includes:

[0064] S22. The chart information in the monitoring configuration includes the page name, chart name, and chart weight;

[0065] like Figure 3 As shown, the monitoring configuration includes page name, chart name, and chart weight, which users can set directly on the web page.

[0066] S23. Categorize the database charts according to the page names to generate categorized database charts;

[0067] like Figure 4 As shown, based on the page name in step S22, database charts with the same page name will be categorized, that is, database charts with the same page name will be converged into one panel. For example, if the page name is: Marketing Platform MKTAPP Monitoring Panel, and the chart names are: Marketing Platform MKTAPP External Call Success Rate, Marketing Platform MKTAPP External Call Failure Rate, and Marketing Platform MKTAPP External Call Average Elapsed Time, then the charts with the database chart names: Marketing Platform MKTAPP External Call Success Rate, Marketing Platform MKTAPP External Call Failure Rate, and Marketing Platform MKTAPP External Call Average Elapsed Time will be categorized into the Marketing Platform MKTAPP Monitoring Panel.

[0068] S24. Sort the categorized database charts according to their status, and place the database charts with abnormal status at the top. When there are multiple categorized database charts with abnormal status, sort the categorized database charts according to their chart weights. When the chart weights are the same, sort the categorized database charts according to their chart names.

[0069] like Figure 4 As shown, categorized database charts are sorted according to their status, with charts in abnormal status placed at the top for easy viewing by relevant personnel. When there are multiple charts in abnormal status, they are sorted according to the chart weights set by the user on the web page. When the chart weights are the same, they are sorted according to their chart names. In this case, the chart names are sorted character by character, starting from the first character. The sorting of chart names can be adjusted according to the actual situation.

[0070] S25. When multiple categorized database charts are in a normal state, the categorized database charts are sorted according to the chart weights. When the chart weights are the same, the categorized database charts are sorted according to the chart names to generate sorted database charts.

[0071] like Figure 4 As shown, for categorized database charts in a normal state, when multiple categorized database charts are in a normal state, they are sorted according to their chart weights. Similarly, when charts have the same weight, they are sorted according to their chart names. In this case, the chart names are sorted sequentially from the first character of the chart name. The sorting of chart names can be adjusted according to the actual situation.

[0072] S26. Return the status of the sorted database chart and the corresponding filtered interface data to the Web page.

[0073] like Figure 4 As shown, after sorting the categorized database charts through steps S24 and S25, the status of the categorized database charts and the corresponding filtered interface data will be returned to the web page to realize the visualization and monitoring of the data.

[0074] Example 2

[0075] Please refer to Figure 5 A data visualization and monitoring device 1 includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, it implements the steps in the above embodiment 1.

[0076] Example 3

[0077] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, it implements the steps in Embodiment 1 above.

[0078] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0079] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0081] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0082] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0083] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0084] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A data visualization and monitoring method, characterized in that, include: The Grafana interface data is collected periodically and in rotation, filtered, and then the filtered interface data is populated into the database chart. Obtain the monitoring configuration of the Web page, obtain the corresponding database chart based on the chart information in the monitoring configuration, and determine whether there is any abnormality in the filtered interface data of the database chart according to the alarm rules in the monitoring configuration. If so, mark the status of the database chart as abnormal; otherwise, mark it as normal, and return the status of the database chart and the corresponding filtered interface data to the Web page. The process of periodically collecting Grafana's interface data, filtering the interface data, and generating filtered interface data includes: The Grafana API data is collected periodically and in rotation. Based on the type of the API data, filtering rules are applied to filter the API data, and filtering results are generated. Determine whether the filtering result conforms to the data retrieval rules. If it does not conform, call the filtering rules according to the type of the filtering result to perform filtering processing until the filtering result conforms to the data retrieval rules. Otherwise, generate filtered interface data based on the filtering result. The step of applying filtering rules to filter the interface data based on its type includes: Determine whether the interface data has multiple layers of nested data. If so, start from the innermost layer of the interface data and proceed outwards to determine the type of each layer of data. Then, recursively call the corresponding filtering rules for each layer of data based on its type to perform filtering processing. Otherwise, directly call the corresponding filtering rules for filtering processing based on the type of the interface data.

2. The data visualization and monitoring method as described in claim 1, characterized in that, The interface data types include arrays and objects. When it is an array, array filtering rules are called for filtering; when it is an object, object filtering rules are called for filtering.

3. The data visualization and monitoring method as described in claim 1, characterized in that, The step of marking the state of the database chart as abnormal also includes: The database charts are color-coded.

4. The data visualization and monitoring method as described in claim 1, characterized in that, The monitoring configuration includes chart information such as page name, chart name, and chart weight. Returning the status of the database chart and the corresponding filtered interface data to the web page includes: The database charts are categorized according to the page names to generate categorized database charts; The categorized database charts are sorted according to their status. Database charts with an abnormal status are placed at the top. When multiple categorized database charts have an abnormal status, they are sorted according to their chart weights. When the chart weights are the same, they are sorted according to their chart names. When multiple categorized database charts are in a normal state, the categorized database charts are sorted according to the chart weight. When the chart weights are the same, the categorized database charts are sorted according to the chart name to generate a sorted database chart. The status of the sorted database chart and the corresponding filtered interface data are returned to the web page.

5. A data visualization and monitoring device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 4.

Images