Monitoring data dynamic adaptation method and device, equipment and storage medium

Abstract

The invention provides a monitoring data dynamic adaptation method and device, equipment and a storage medium. The method comprises the following steps: collecting monitoring data of at least one theme, and storing monitoring data of different themes into different queues; for the monitoring data in each queue, obtaining target monitoring configuration corresponding to the monitoring data through a pre-constructed configuration database; performing data preprocessing on the monitoring data according to the target data index, and calculating the preprocessed data according to the target index algorithm set to obtain a monitoring data index value; and aggregating the monitoring data index values according to a target aggregation rule to obtain an index aggregation value, judging whether the index aggregation value accords with a preset alarm rule, and if so, pushing alarm information. According to the method, the preset configuration database is combined with the distributed message queue, automatic monitoring index adaptation and monitoring data processing are carried out on the monitoring data of different themes, and the method is suitable for multiple services, efficient, rapid and high in accuracy.

Description

Technical Field

[0001] This application relates to the field of data monitoring technology, and in particular to a method, apparatus, device, and storage medium for dynamic adaptation of monitoring data. Background Technology

[0002] In today's era of big data and cloud computing, the demand for monitoring physical devices and cloud resources is constantly growing. As enterprises expand and business complexity increases, monitoring systems need to be able to monitor and analyze massive amounts of data metrics in real time to ensure system stability and performance optimization. Simultaneously, the requirements for monitoring are also rising; enterprises need monitoring systems that can comprehensively and accurately reflect the system's operational status, promptly detect anomalies, and take appropriate measures. The processing of various complex metrics presents significant challenges, requiring monitoring systems to possess powerful data processing and analysis capabilities, effectively filtering, aggregating, and analyzing massive amounts of data metrics to extract valuable information and insights. This means processing a vast and complex amount of information, including numerous metrics of different types and sources. Therefore, monitoring systems need to integrate and analyze data metrics from different sources, while also possessing flexibility and scalability to adapt to constantly changing monitoring needs and business environments.

[0003] Existing monitoring data adaptation technologies require manual collection, organization, and analysis of corresponding metrics for each business, followed by appropriate monitoring configuration based on these metrics. Due to the complexity of the metrics, the calculation process becomes extremely complex and difficult. Furthermore, the diversity and complexity of the data, along with limitations in the calculation methods, can easily lead to errors or biases, resulting in inaccurate calculation results. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for dynamic adaptation of monitoring data, which solves the problem that existing monitoring data adaptation technologies can only target specific business analysis indicators and require manual monitoring configuration, which can easily lead to errors or deviations in the calculation process and consume a lot of manpower.

[0005] Firstly, this application provides a method for dynamic adaptation of monitoring data, including:

[0006] Collect monitoring data for at least one theme and store the monitoring data for different themes into different queues;

[0007] For the monitoring data in each queue, the target monitoring configuration corresponding to the monitoring data is obtained through a pre-built configuration database. The target monitoring configuration includes target data indicators, target indicator algorithm set and target aggregation rules. The target index algorithm set includes at least one target index algorithm. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data topic.

[0008] The monitoring data is preprocessed according to the target data indicators, and the preprocessed data is calculated according to the target indicator algorithm set to obtain the monitoring data indicator values.

[0009] The monitoring data indicator values ​​are aggregated according to the target aggregation rules to obtain the indicator aggregate value. It is then determined whether the indicator aggregate value meets the preset alarm rules. If so, an alarm information is pushed.

[0010] Optionally, in the method described above, the monitoring data includes data message configuration information, the message configuration information includes a configuration database, and storing monitoring data for different topics into different queues includes:

[0011] Construct a message middleware, which includes several message topics, each message topic includes at least one message queue, and each message queue is matched with a corresponding configuration database;

[0012] The collected monitoring data is transmitted to a matching message topic, and the monitoring data is stored in a message queue that matches the monitoring data configuration database in each message topic.

[0013] Optionally, in the method described above, obtaining the target monitoring configuration corresponding to the monitoring data through a pre-built configuration database includes:

[0014] Determine the target topic of the monitoring data, and obtain the target monitoring configuration that is compatible with the target topic based on the adaptation relationship between the monitoring configuration and the monitoring data topic.

[0015] Optionally, in the method described above, the step of preprocessing the monitoring data according to the target data indicator includes:

[0016] The monitoring data is cleaned to obtain cleaned monitoring data;

[0017] Determine the data filtering criteria based on the target data indicators;

[0018] Generate query statements corresponding to the data filtering conditions using a data query language;

[0019] The cleaned monitoring data is queried using the query statement, and the data obtained from the query is used as the preprocessed data.

[0020] Optionally, in the method described above, the step of calculating the monitoring data indicator value based on the preprocessed data according to the target indicator algorithm set includes:

[0021] For each target indicator algorithm in the target algorithm set, an algorithm factor is determined. The algorithm factor is the smallest unit of the numerator or denominator in the target indicator algorithm formula corresponding to the monitoring indicator.

[0022] The algorithm factors of each target index algorithm are matched with the preprocessed data to generate the calculation task of the target index algorithm.

[0023] The calculation task is executed to obtain the monitoring data index values ​​corresponding to each target index algorithm.

[0024] Optionally, in the method described above, the step of aggregating the monitoring data indicator values ​​according to the target aggregation rule to obtain the indicator aggregated value includes:

[0025] For each target indicator algorithm, generate different aggregation tasks based on the monitoring data indicator values.

[0026] Each aggregation task is executed according to the target aggregation rule to obtain the aggregated value of the target index algorithm.

[0027] Optionally, the method described above, before obtaining the target monitoring configuration corresponding to the monitoring data through a pre-built configuration database, further includes:

[0028] Determine the data structure of the monitoring data, wherein the data structure includes data fields and data values;

[0029] The data field is used as the key of the JSON object, and the data value is used as the value of the JSON object. The JSON object is populated with monitoring data to obtain the formatted monitoring data.

[0030] Secondly, this application provides a monitoring data dynamic adaptation device, comprising:

[0031] The data acquisition module is used to collect monitoring data for at least one theme and store the monitoring data for different themes into different queues.

[0032] The dynamic adaptation module is used to obtain the target monitoring configuration corresponding to the monitoring data in each queue through a pre-built configuration database. The target monitoring configuration includes target indicator algorithms and target aggregation rules. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data topic.

[0033] The indicator filtering module is used to preprocess the monitoring data and calculate the monitoring data indicator value according to the target indicator algorithm.

[0034] The aggregation module is used to aggregate monitoring data indicator values ​​according to target aggregation rules to obtain indicator aggregate values, and to determine whether the indicator aggregate values ​​meet preset alarm rules. If so, alarm information is pushed.

[0035] Thirdly, this application provides an electronic device, including a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-executable instructions to implement the monitoring data dynamic adaptation method described in any one of the first aspects above.

[0036] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the monitoring data dynamic adaptation method described in any one of the first aspects above.

[0037] Fifthly, this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the monitoring data dynamic adaptation method described in any of the first aspects above.

[0038] The monitoring data dynamic adaptation method, apparatus, device, and storage medium provided in this application collect monitoring data for at least one theme and store the monitoring data for different themes in different queues. For the monitoring data in each queue, the target monitoring configuration corresponding to the monitoring data is obtained through a pre-built configuration database. The target monitoring configuration includes target data indicators, a target indicator algorithm set, and target aggregation rules. The target index algorithm set includes at least one target index algorithm. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data theme. The monitoring data is preprocessed according to the target data indicators, and the preprocessed data is calculated according to the target indicator algorithm set to obtain monitoring data indicator values. The monitoring data indicator values ​​are aggregated according to the target aggregation rules to obtain indicator aggregation values. It is then determined whether the indicator aggregation values ​​meet the preset alarm rules. If so, alarm information is pushed. This application abstracts and organizes various indicators in advance to form a preset configuration database. Using a distributed message queue, it automatically adapts monitoring indicators and processes monitoring data for different themes. It is applicable to multiple businesses, efficient, fast, and highly accurate. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 This is a schematic diagram illustrating an application scenario of the dynamic adaptation method for monitoring data provided in this application embodiment.

[0041] Figure 2 A flowchart of the dynamic adaptation method for monitoring data provided in the embodiments of this application.

[0042] Figure 3 This is a schematic diagram of a monitoring data dynamic adaptation device provided in an embodiment of this application.

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for a dynamic adaptation method of monitoring data, provided in an embodiment of this application.

[0044] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0047] In relevant data monitoring and processing technologies, it is necessary to analyze and collect indicators for each business, and then configure the monitoring data accordingly based on the indicators. Due to the complexity of the indicators, the calculation process becomes extremely complex and difficult. Furthermore, in this process, errors or deviations may easily occur due to factors such as the diversity and complexity of the data and the limitations of the calculation methods, leading to inaccurate calculation results.

[0048] To address the aforementioned technical issues, this application aims to propose a method, apparatus, device, and storage medium for dynamic adaptation of monitoring data. The main concept of this method is to automatically adapt monitoring indicators and process monitoring data for different topics by combining a preset configuration database with a distributed message queue. This method is applicable to multiple businesses, is efficient and fast, and has high accuracy.

[0049] To better understand the solutions of the embodiments of this application, an application scenario involved in the embodiments of this application will be introduced below.

[0050] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the dynamic adaptation method for monitoring data provided in the embodiments of this application, such as... Figure 1As shown, the system includes a client 100, a server 200, and a data source 300. The client 100 can send data monitoring-related commands to the server 200 and receive monitoring and alarm information from the server 200. The data source 300 can include network devices in a cloud computing environment, applications deployed in a cloud computing environment, etc. The data source 300 generates various monitoring data, such as request response time, database query performance, error logs, network traffic, etc. The server 200 includes a pre-configured configuration database, a data acquisition module, a monitoring adaptation module, and a data processing module. The data acquisition module can collect monitoring data from the data source 300 and store monitoring data on different topics into different queues in a message middleware. The configuration database stores pre-configured monitoring settings. The monitoring adaptation module in server 200 dynamically creates monitoring processing programs based on the monitoring data theme and other information, including monitoring data message configuration, monitoring data information configuration, and monitoring indicator algorithm configuration. The monitoring adaptation module pulls monitoring data from the message middleware, converts the monitoring data into JSON format, matches the target monitoring configuration corresponding to the monitoring data theme, and performs indicator data filtering, indicator value calculation, and aggregation processing on the monitoring data according to the target monitoring configuration. Finally, the processed indicator aggregation value is stored, and alarm judgment is performed based on the indicator aggregation value. If the indicator aggregation value meets the preset alarm rules, alarm information is pushed.

[0051] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0052] Figure 2 A flowchart illustrating the dynamic adaptation method for monitoring data provided in this application embodiment. Figure 2 As shown, the method in this embodiment includes:

[0053] S201: Collect monitoring data for at least one theme and store the monitoring data for different themes into different queues.

[0054] The execution entity of this application embodiment can be a server or a monitoring data dynamic adaptation system in the server, wherein the monitoring data dynamic adaptation system can be implemented by software.

[0055] Understandably, in monitoring data, a topic typically refers to the subject or topic identifier of the data, used to identify the specific category or theme to which the data belongs. Monitoring data topics can be categorized and identified based on monitoring objects, monitoring metrics, monitoring scenarios, etc., to facilitate data organization, management, and analysis. When storing monitoring data, it can be categorized according to its topic, with data from different topics stored in corresponding queues. This ensures that each queue correctly stores the monitoring data for its corresponding topic, achieving data classification, management, and processing.

[0056] S202: For the monitoring data in each queue, the target monitoring configuration corresponding to the monitoring data is obtained through a pre-built configuration database. The target monitoring configuration includes target data indicators, target indicator algorithm set and target aggregation rules. The target index algorithm set includes at least one target index algorithm. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data topic.

[0057] In this step, the pre-built configuration database can include multiple pre-configured monitoring configurations adapted to different monitoring data themes. This allows the monitoring data of each queue to be quickly adapted to its corresponding monitoring configuration based on its theme. When processing the monitoring data, the corresponding monitoring metrics can be dynamically applied without the need for manual re-analysis of the data, thereby confirming the configuration information such as monitoring metrics and metric algorithms.

[0058] S203: Perform data preprocessing on the monitoring data according to the target data indicators, and calculate the preprocessed data according to the target indicator algorithm set to obtain the monitoring data indicator value.

[0059] In this step, preprocessing operations such as data filtering can be performed on the monitoring data based on the target data indicators to obtain effective monitoring data and ensure its accuracy. Then, different target indicator algorithms are used for calculation. These target data indicators can include instantaneous speed, flow rate, etc., and the algorithm is the method used to calculate these indicator values. It can be simple addition, subtraction, multiplication, and division operations, or more complex statistical analysis or machine learning algorithms, thereby obtaining accurate monitoring data indicator values ​​and achieving effective monitoring and management of the monitoring system.

[0060] S204: Aggregate the monitoring data indicator values ​​according to the target aggregation rules to obtain the indicator aggregation value, and determine whether the indicator aggregation value meets the preset alarm rules. If so, push the alarm information.

[0061] In this step, the collected monitoring data metric values ​​are calculated according to the determined target aggregation rules to obtain aggregated values. These aggregated values ​​are recorded and analyzed for subsequent monitoring and decision-making. Furthermore, the monitoring data metric values ​​can be updated periodically based on actual conditions, and the aggregation rules and calculation methods can be adjusted as needed to ensure accurate and useful aggregated values. Administrators can predefine alarm rules as needed, such as "alarm when network traffic exceeds a preset first threshold," allowing them to determine whether to trigger alarm rules based on aggregated metric values. If an aggregated metric value does not trigger an alarm rule, it can be stored for administrators to review.

[0062] The dynamic adaptation method for monitoring data provided in this embodiment collects monitoring data for at least one theme and stores the monitoring data for different themes in different queues. For the monitoring data in each queue, the target monitoring configuration corresponding to the monitoring data is obtained through a pre-built configuration database. The target monitoring configuration includes target data indicators, a target indicator algorithm set, and target aggregation rules. The target index algorithm set includes at least one target index algorithm. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data theme. The monitoring data is preprocessed according to the target data indicators, and the preprocessed data is calculated according to the target indicator algorithm set to obtain monitoring data indicator values. The monitoring data indicator values ​​are aggregated according to the target aggregation rules to obtain indicator aggregation values. It is then determined whether the indicator aggregation values ​​meet the preset alarm rules. If so, alarm information is pushed. This application abstracts and organizes various indicators in advance to form a preset configuration database. Using a distributed message queue, it automatically adapts monitoring indicators and processes monitoring data for different themes. It is applicable to multiple businesses, efficient, fast, and highly accurate.

[0063] The technical solution for the above-mentioned dynamic adaptation method of monitoring data is described in detail below.

[0064] In one possible implementation, the monitoring data includes data message configuration information, and the message configuration information includes a configuration database. The monitoring data dynamic adaptation method provided in this embodiment constructs different topics in the message middleware, sets different message queues in the topics, and stores the monitoring data in the message queue that matches the monitoring data configuration database.

[0065] Specifically, storing monitoring data for different topics into different queues includes: constructing a message middleware, wherein the message middleware includes several message topics, each message topic includes at least one message queue, and each message queue is matched with a corresponding configuration database; transmitting the collected monitoring data to the matched message topic, and storing the monitoring data in each message topic into a message queue that matches the monitoring data configuration database.

[0066] Understandably, message middleware can act as a decoupling mechanism between system components. After monitoring data is sent to the message middleware, different system components can subscribe to and process this data as needed, thereby achieving loose coupling and improving system flexibility and maintainability. Furthermore, by sending monitoring data to the message middleware, asynchronous processing can be achieved, meaning that the generation and processing of monitoring data are separated. This improves system response speed and throughput, preventing performance degradation due to monitoring data processing blocking.

[0067] In this application, message processing can be implemented using Kafka. When monitoring data needs to be pulled from the message middleware, the consumer can periodically or as needed pull messages from the message middleware. Once the consumer has pulled the monitoring data, it can process this data.

[0068] In this embodiment, by constructing different topics in the message middleware and setting different message queues in the topics, the monitoring data is stored in the message queue that matches the monitoring data configuration database, thereby improving the system's performance, reliability, and flexibility.

[0069] In one possible implementation, the dynamic adaptation method for monitoring data provided in this embodiment determines the target monitoring configuration of the monitoring data by means of the adaptation relationship between the monitoring configuration and the monitoring data topic.

[0070] Specifically, the target monitoring configuration corresponding to the monitoring data is obtained through a pre-built configuration database, including: determining the target topic of the monitoring data, and obtaining the target monitoring configuration that is compatible with the target topic based on the adaptation relationship between the monitoring configuration and the monitoring data topic.

[0071] Understandably, administrators can create a configuration database to store at least one monitoring configuration and its compatibility with monitoring data topics. This compatibility can be one-to-one or one-to-many, and can be stored using relational databases (such as MySQL or PostgreSQL) or NoSQL databases (such as MongoDB). Monitoring configurations are defined in the configuration database, including monitoring parameters, thresholds, rules, and other information. Each monitoring configuration must have a unique identifier for subsequent retrieval and matching. When it is necessary to obtain the target monitoring configuration corresponding to the monitoring data, the compatibility relationship can be retrieved by querying the configuration database. For example, based on the characteristics or identifiers of the monitoring data, the corresponding monitoring configuration information can be found. In other words, in this embodiment, based on the target topic of the monitoring data, the expected target monitoring configuration can be found in the configuration database.

[0072] In this embodiment, the target monitoring configuration of the monitoring data is determined by the adaptation relationship between the monitoring configuration and the monitoring data topic, thereby realizing dynamic adaptation of the monitoring data configuration. This allows for more effective management and application of the monitoring configuration, improving the efficiency and reliability of the monitoring system.

[0073] In one possible implementation, the monitoring data dynamic adaptation method provided in this embodiment cleans the monitoring data and determines the data filtering conditions according to the target data indicators, thereby filtering the monitoring data and obtaining preprocessed data.

[0074] Specifically, the monitoring data is preprocessed according to the target data indicators, including: cleaning the monitoring data to obtain cleaned monitoring data; determining data filtering conditions according to the target data indicators; generating query statements corresponding to the data filtering conditions using a data query language; querying the cleaned monitoring data using the query statements, and using the data obtained from the query as the preprocessed data.

[0075] It's understandable that monitoring data includes various types of information. However, not all of this data is useful for data monitoring; in fact, it may cause data bias. Therefore, precise filtering of indicator data is necessary. The filtering process begins by determining the filtering criteria based on the target data indicator. For example, this can be determined by the indicator type, or by combining the indicator type with a preset threshold. For instance, if the target data indicator is battery level, then data with battery level can be filtered based on this indicator. Alternatively, data with a preset threshold of "greater than 50%" can be filtered to include data with battery level greater than 50%. Specifically, corresponding query statements or code can be written to filter the data. For example, if using SQL for data filtering, a query statement like "SELECT * FROM table WHERE battery_level>50" can be generated based on the filtering criteria.

[0076] In this embodiment, the monitoring data is cleaned and the data filtering conditions are determined according to the target data indicators, thereby filtering the monitoring data and obtaining pre-processed data, which improves the effectiveness of the monitoring data and facilitates its further utilization in subsequent steps.

[0077] In one possible implementation, the monitoring data dynamic adaptation method provided in this embodiment determines the algorithm factor for each target indicator algorithm, then generates the calculation task corresponding to each target indicator algorithm by combining the preprocessed data, thereby obtaining the monitoring data indicator value corresponding to each target indicator algorithm, and obtaining the indicator aggregation value corresponding to each target indicator algorithm according to the monitoring data indicator value and the target aggregation rule.

[0078] Specifically, the monitoring data index value is obtained by calculating the preprocessed data according to the target index algorithm set, including: determining the algorithm factor for each target index algorithm in the target algorithm set, wherein the algorithm factor is the smallest component of the numerator or denominator in the target index algorithm formula corresponding to the monitoring index; matching the algorithm factor of each target index algorithm with the preprocessed data to generate a calculation task for the target index algorithm; and executing the calculation task to obtain the monitoring data index value corresponding to each target index algorithm.

[0079] It is understandable that target indicator algorithms can include: ① custom algorithms that support operations such as addition, subtraction, multiplication, division, and parentheses, e.g., (total-used) / total; ② accumulated value ≥ instantaneous value; ③ accumulated value ≥ instantaneous value ≥ rate (instantaneous value divided by time); ④ instantaneous value ≥ rate, etc. Using the preprocessed data as algorithm factors, the monitoring data indicator values ​​corresponding to each target indicator algorithm can be obtained.

[0080] Furthermore, the monitoring data indicator values ​​are aggregated according to the target aggregation rules to obtain the indicator aggregate values, including: generating different aggregation tasks for the monitoring data indicator values ​​corresponding to each target indicator algorithm; executing each aggregation task according to the target aggregation rules to obtain the indicator aggregate values ​​corresponding to each target indicator algorithm.

[0081] Understandably, the target aggregation rules determine the type of aggregation to be performed on the monitoring data metric values, such as summation, average, maximum, minimum, etc. Based on the determined target aggregation rules, the collected monitoring data metric values ​​are calculated accordingly to obtain aggregated values. The calculated aggregated metric values ​​can also be recorded and analyzed for subsequent monitoring and decision-making. Furthermore, the monitoring data metric values ​​are updated regularly based on actual conditions, and the aggregation rules and calculation methods are adjusted as needed to ensure accurate and useful aggregated metric values ​​are obtained.

[0082] In this embodiment, algorithm factors are determined for each target indicator algorithm, and then calculation tasks corresponding to each target indicator algorithm are generated by combining the preprocessed data, thereby obtaining the monitoring data indicator values ​​corresponding to each target indicator algorithm. Based on the monitoring data indicator values ​​and target aggregation rules, the indicator aggregation values ​​corresponding to each target indicator algorithm are obtained. This enables efficient and accurate configuration and processing of monitoring data, and the obtained processed indicator aggregation values ​​that can provide feedback on whether the monitoring data is abnormal.

[0083] In one possible implementation, the dynamic adaptation method for monitoring data provided in this embodiment, before obtaining the target monitoring configuration corresponding to the monitoring data through a pre-built configuration database, further includes: determining the data structure of the monitoring data, the data structure including data fields and data values; using the data fields as keys of a JSON object, using the data values ​​as values ​​of a JSON object, and filling the JSON object with the monitoring data to obtain the format-converted monitoring data.

[0084] Understandably, JSON (JavaScript Object Notation) is a lightweight data exchange format suitable for transmitting data over networks. Formatting monitoring data into JSON can reduce data transmission costs and latency. Formatting monitoring data into a uniform JSON format helps ensure data consistency and standardization, making the data easier to understand and process. Furthermore, JSON data is easy to parse and analyze, which can help monitoring systems perform real-time monitoring, alerting, and analysis, thereby enabling timely problem detection and appropriate action.

[0085] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0086] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0087] Figure 3 This is a schematic diagram of a dynamic monitoring data adaptation device provided in an embodiment of this application. Figure 3 As shown, the monitoring data dynamic adaptation device includes:

[0088] The acquisition module 31 is used to acquire monitoring data for at least one theme and store the monitoring data for different themes into different queues.

[0089] The dynamic adaptation module 32 is used to obtain the target monitoring configuration corresponding to the monitoring data in each queue through a pre-built configuration database. The target monitoring configuration includes target indicator algorithms and target aggregation rules. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data topic.

[0090] The indicator filtering module 33 is used to preprocess the monitoring data and calculate the preprocessed data according to the target indicator algorithm to obtain the monitoring data indicator value.

[0091] The aggregation module 34 is used to aggregate the monitoring data indicator values ​​according to the target aggregation rules to obtain the indicator aggregation value, and to determine whether the indicator aggregation value meets the preset alarm rules. If so, an alarm information is pushed.

[0092] In one possible design, the monitoring data includes data message configuration information, which includes a configuration database. The acquisition module 31 is specifically used for:

[0093] Construct a message middleware, which includes several message topics, each message topic includes at least one message queue, and each message queue is matched with a corresponding configuration database;

[0094] The collected monitoring data is transmitted to a matching message topic, and the monitoring data is stored in a message queue that matches the monitoring data configuration database in each message topic.

[0095] In one possible design, the dynamic adaptation module 32 is specifically used for:

[0096] Determine the target topic of the monitoring data, and obtain the target monitoring configuration that is compatible with the target topic based on the adaptation relationship between the monitoring configuration and the monitoring data topic.

[0097] In one possible design, the indicator filtering module 33 is specifically used for:

[0098] The monitoring data is cleaned to obtain cleaned monitoring data;

[0099] Determine the data filtering criteria based on the target data indicators;

[0100] Generate query statements corresponding to the data filtering conditions using a data query language;

[0101] The cleaned monitoring data is queried using the query statement, and the data obtained from the query is used as the preprocessed data.

[0102] In one possible design, the indicator filtering module 33 is specifically used for:

[0103] For each target indicator algorithm in the target algorithm set, an algorithm factor is determined. The algorithm factor is the smallest unit of the numerator or denominator in the target indicator algorithm formula corresponding to the monitoring indicator.

[0104] The algorithm factors of each target index algorithm are matched with the preprocessed data to generate the calculation task of the target index algorithm.

[0105] The calculation task is executed to obtain the monitoring data index values ​​corresponding to each target index algorithm.

[0106] In one possible design, the aggregation module 34 is specifically used for:

[0107] For each target indicator algorithm, generate different aggregation tasks based on the monitoring data indicator values.

[0108] Each aggregation task is executed according to the target aggregation rule to obtain the aggregated value of the target index algorithm.

[0109] In one possible design, the dynamic adaptation module 32 is also specifically used for:

[0110] Determine the data structure of the monitoring data, wherein the data structure includes data fields and data values;

[0111] The data field is used as the key of the JSON object, and the data value is used as the value of the JSON object. The JSON object is populated with monitoring data to obtain the formatted monitoring data.

[0112] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0113] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0114] Figure 4 This is a schematic diagram of the structure of an electronic device suitable for a dynamic adaptation method of monitoring data, provided in an embodiment of this application. Figure 4 As shown, the electronic device of this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown) a processor, a memory 41, and a computer program stored in the memory 41 that can run on at least one processor 40, which executes the computer program to implement the steps in any of the above method embodiments.

[0115] The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0116] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0117] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0118] In some embodiments, memory 41 may be an internal storage unit of an electronic device, such as the memory of the electronic device. In other embodiments, memory 41 may be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 41 may include both internal and external storage units of the electronic device. Memory 41 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 41 can also be used to temporarily store data that has been output or will be output.

[0119] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0120] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0121] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the aforementioned electronic device.

[0122] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0123] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0124] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0125] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for dynamic adaptation of monitoring data, characterized in that, include: Collect monitoring data for at least one theme and store the monitoring data for different themes into different queues; For the monitoring data in each queue, the target monitoring configuration corresponding to the monitoring data is obtained through a pre-built configuration database. The target monitoring configuration includes target data indicators, target indicator algorithm set and target aggregation rules. The target index algorithm set includes at least one target index algorithm. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data topic. The monitoring data is preprocessed according to the target data indicators, and the preprocessed data is calculated according to the target indicator algorithm set to obtain the monitoring data indicator values. The monitoring data indicator values ​​are aggregated according to the target aggregation rules to obtain the indicator aggregate value. It is then determined whether the indicator aggregate value meets the preset alarm rules. If so, an alarm information is pushed.

2. The method according to claim 1, characterized in that, The monitoring data includes data message configuration information, which includes a configuration database. Storing monitoring data for different topics into different queues includes: Construct a message middleware, which includes several message topics, each message topic includes at least one message queue, and each message queue is matched with a corresponding configuration database; The collected monitoring data is transmitted to a matching message topic, and the monitoring data is stored in a message queue that matches the monitoring data configuration database in each message topic.

3. The method according to claim 1, characterized in that, The step of obtaining the target monitoring configuration corresponding to the monitoring data through a pre-built configuration database includes: Determine the target topic of the monitoring data, and obtain the target monitoring configuration that is compatible with the target topic based on the adaptation relationship between the monitoring configuration and the monitoring data topic.

4. The method according to claim 1, characterized in that, The step of preprocessing the monitoring data according to the target data indicators includes: The monitoring data is cleaned to obtain cleaned monitoring data; Determine the data filtering criteria based on the target data indicators; Generate query statements corresponding to the data filtering conditions using a data query language; The cleaned monitoring data is queried using the query statement, and the data obtained from the query is used as the preprocessed data.

5. The method according to claim 1, characterized in that, The step of calculating the monitoring data indicator values ​​based on the target indicator algorithm set after preprocessing the data includes: For each target indicator algorithm in the target algorithm set, an algorithm factor is determined. The algorithm factor is the smallest unit of the numerator or denominator in the target indicator algorithm formula corresponding to the monitoring indicator. The algorithm factors of each target index algorithm are matched with the preprocessed data to generate the calculation task of the target index algorithm. The calculation task is executed to obtain the monitoring data index values ​​corresponding to each target index algorithm.

6. The method according to claim 5, characterized in that, The aggregation of monitoring data indicator values ​​according to the target aggregation rule to obtain the indicator aggregated value includes: For each target indicator algorithm, generate different aggregation tasks based on the monitoring data indicator values. Each aggregation task is executed according to the target aggregation rule to obtain the aggregated value of the target index algorithm.

7. The method according to claim 1, characterized in that, Before obtaining the target monitoring configuration corresponding to the monitoring data through a pre-built configuration database, the method further includes: Determine the data structure of the monitoring data, wherein the data structure includes data fields and data values; The data field is used as the key of the JSON object, and the data value is used as the value of the JSON object. The JSON object is populated with monitoring data to obtain the formatted monitoring data.

8. A dynamic adaptation device for monitoring data, characterized in that, include: The data acquisition module is used to collect monitoring data for at least one theme and store the monitoring data for different themes into different queues. The dynamic adaptation module is used to obtain the target monitoring configuration corresponding to the monitoring data in each queue through a pre-built configuration database. The target monitoring configuration includes target indicator algorithms and target aggregation rules. The configuration database is used to store at least one monitoring configuration and the adaptation relationship between each monitoring configuration and the monitoring data topic. The indicator filtering module is used to preprocess the monitoring data and calculate the monitoring data indicator value according to the target indicator algorithm. The aggregation module is used to aggregate monitoring data indicator values ​​according to target aggregation rules to obtain indicator aggregate values, and to determine whether the indicator aggregate values ​​meet preset alarm rules. If so, alarm information is pushed.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images