Monitoring index adaptive whole-process processing method and system

Abstract

The application belongs to the technical field of data processing. A monitoring index adaptive whole-process processing method and system are provided. Monitoring index metadata of a collection object is acquired, and the monitoring index metadata is stored after being formatted. After each interval of a set time period, the stored data is updated according to newly collected monitoring index metadata. The updated index metadata is normalized, and the normalized index metadata is subjected to grouping mutation point detection. For a data group in which no mutation point is identified, compression storage is performed according to the ratio of storage frequency to collection frequency. For a data group in which a mutation point is identified, compression storage is performed according to the ratio of storage frequency to collection frequency and a set number of data points before and after the mutation point. The storage efficiency and data fidelity are effectively balanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a monitoring index adaptive whole-process processing method and system. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present application and do not necessarily constitute prior art.

[0003] At present, the Internet of Things technology is developing rapidly, and the monitoring system is facing unprecedented technical challenges and innovation opportunities. With the deep application of new generation information technologies such as 5G and edge computing, modern monitoring requirements have expanded from traditional hardware performance indicators (such as CPU, memory, disk and other basic resources) to comprehensive monitoring systems covering middleware, database, operating system and other whole stack components, and also need to collect API response delay, message queue depth, file system capacity and other business critical indicators. The diversification of monitoring dimensions makes the traditional monitoring scheme based on static configuration difficult to cope with dynamic changes in metadata, integration of heterogeneous data sources and massive time series data storage.

[0004] The existing scheme has the following problems: the response to metadata changes is lagging, and the structural changes of the monitoring object cannot be sensed in time; the data collection method is rigid and cannot adapt to the characteristics of different monitoring objects; the storage efficiency and data fidelity are difficult to balance, and often one is lost for the other, especially in the quasi-real-time monitoring scene using the pull mode (typical sampling interval 10 seconds to several minutes). Although the traditional long storage window design effectively reduces the storage overhead, it is easy to lose transient abnormal data (such as sudden jitter, instantaneous traffic peak, etc.) within the window, and this data distortion may cause serious operation and maintenance blind area, and even lead to misdiagnosis of fault diagnosis. SUMMARY

[0005] In order to solve the problems of the prior art, the present application provides a monitoring index adaptive whole-process processing method and system, which realizes the whole-process automatic processing of the collection, processing and storage of monitoring index metadata. For data groups without identified mutation points, the data groups are compressed and stored according to the ratio of storage frequency to collection frequency, and for data groups with identified mutation points, the data groups are compressed and stored according to the ratio of storage frequency to collection frequency and the number of data points set before and after the mutation point. The storage efficiency and data fidelity are effectively balanced, and the data processing efficiency is improved.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] In the first aspect, the present application provides a monitoring index adaptive whole-process processing method.

[0008] A monitoring indicator adaptive full-process processing method includes the following steps:

[0009] Obtain the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period;

[0010] The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0011] In one implementation of the first aspect of the present invention, the monitoring indicator metadata is formatted and stored, and after each set period, the stored data is updated based on the newly collected monitoring indicator metadata, including:

[0012] Subscribe to the metadata processing queue, obtain the monitoring indicator metadata of the collection object according to the collection data object type and indicator collection address, format the metadata, and record it in the first setting file;

[0013] After each set period, the monitoring indicator metadata of the collection object is obtained again and written into a new second setting file. The first setting file and the second setting file are compared. If the monitoring indicator metadata has not changed, the second setting file is deleted. If there is a difference, the difference metadata is extracted and the second setting file is used to overwrite the first setting file. The difference items are sent to the preset client for data change reminders.

[0014] As a further limitation of the first aspect of the present invention, obtaining the monitoring indicator metadata of the collection object again after each set period of time also includes the following process:

[0015] Determine whether the monitoring indicator metadata in the second setting file is generated for the first time: If it is generated for the first time, synchronize the monitoring indicator metadata directly to the time series database; if it is not generated for the first time, compare it with the monitoring indicator metadata in the first setting file.

[0016] In an implementation of the first aspect of the present invention, identifying the mutation point of each data group by median absolute deviation includes: ,in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median;

[0017] When data points When it is greater than the set threshold, the The data points are mutation points.

[0018] In one implementation of the first aspect of the present invention, for a data set in which no mutation point is identified, the number of retained data points is: , N represents the total number of data points in the data group, Represents the storage frequency, Represents the acquisition frequency.

[0019] In one implementation of the first aspect of the present invention, for a data set in which a mutation point is identified, the number of retained data points is: , where N represents the total number of data points in the data set, Represents the storage frequency, represents the acquisition frequency, Represents the number of mutation points, It means retaining each mutation point as well as the first and last two data points of each mutation point.

[0020] In an implementation of the first aspect of the present invention, before obtaining the monitoring indicator metadata of the collection object, the following process is also included:

[0021] After configuring the monitoring indicator object and enabling the collection configuration, the collected data will be automatically pushed to the monitoring indicator metadata processing queue. The monitoring indicator metadata for processing can be extracted by subscribing to the monitoring indicator metadata processing queue.

[0022] In one implementation of the first aspect of the present invention, a thread double buffering setting is adopted, in which the Active Buffer is used to receive new real-time data, and the Ready Buffer is used to process window data. This can effectively prevent data from being locked. After the buffers are exchanged according to the storage frequency (data in the Active Buffer enters the Ready Buffer), the data is processed.

[0023] In a second aspect, the present invention provides a monitoring indicator adaptive full-process processing system.

[0024] A monitoring indicator adaptive full-process processing system, comprising:

[0025] The data acquisition unit is configured to: acquire monitoring indicator metadata of the acquisition object, format the monitoring indicator metadata and store it, and update the stored data according to the newly acquired monitoring indicator metadata after each set period;

[0026] The compression storage unit is configured to: normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store the data group for which no mutation point is identified according to the ratio of storage frequency to acquisition frequency, and compress and store the data group for which mutation point is identified according to the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point.

[0027] In a third aspect, the present invention provides a computer device comprising: a processor and a computer-readable storage medium;

[0028] a processor adapted to execute a computer program;

[0029] A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the monitoring indicator adaptive full-process processing method as described in the first aspect of the present invention.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention innovatively proposes an adaptive full-process processing method for monitoring indicators, which realizes the full-process automated processing of the collection, processing and storage of monitoring indicator metadata. For data groups without identified mutation points, compression storage is performed based on the ratio of storage frequency to acquisition frequency. For data groups with identified mutation points, compression storage is performed based on the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point. This effectively takes into account both storage efficiency and data fidelity, and improves the efficiency of data processing.

[0032] 2. The present invention innovatively proposes a full-process adaptive processing method for monitoring indicators. Metadata is formatted and recorded in a first setting file. After each set period, the monitoring indicator metadata of the collection object is obtained again and written to a new second setting file. The first setting file and the second setting file are compared. If the monitoring indicator metadata has not changed, the second setting file is deleted. If there is a difference, the difference metadata is extracted and the second setting file is used to overwrite the first setting file. The difference items are sent to the preset client as a data change reminder. Changes in the monitoring indicator of the monitoring object (such as the addition / deletion of indicators) are automatically detected without manual maintenance of the collection template. Through the difference comparison and user confirmation process, the accuracy and flexibility of metadata updates are ensured, and monitoring interruptions caused by version upgrades are avoided.

[0033] 3. The present invention innovatively proposes an adaptive full-process processing method for monitoring indicators, normalizes the updated indicator metadata, supports multiple heterogeneous data formats such as Prometheus, JSON, key-value pairs, etc., and only requires configuration when adding new monitoring objects without the need to develop adaptation code. It supports multiple collection methods such as API pull, TCP, UDP monitoring, and adapts to the complex environment of the Internet of Things.

[0034] 4. This invention innovatively proposes an adaptive full-process processing method for monitoring indicators. For stable data groups, it downsamples proportionally to reduce redundant storage. For data groups with mutation points, it accurately identifies anomalies through an improved Hampel filtering algorithm and expands the storage context (retaining the "first 1 and last 2" data points). By combining block submission of data with breakpoint retry, it ensures low data loss.

[0035] 5. The present invention innovatively proposes an adaptive full-process processing method for monitoring indicators, which adopts double-buffered high-frequency sampling. The bottom layer collects data at a high frequency (such as 1 second / time), and the upper layer downsamples and stores it on demand, which not only avoids missing transient anomalies but also controls storage expansion. At the same time, the storage granularity is dynamically adjusted based on data fluctuations, with high fidelity for key data and high compression for non-key data, thereby improving processing efficiency.

[0036] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0038] Figure 1 A schematic diagram of a data processing flow provided for an exemplary embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a process for monitoring indicator metadata management provided by an exemplary embodiment of the present invention;

[0040] Figure 3 A schematic diagram of normalization processing of monitoring indicator data provided by an exemplary embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a double buffer processing mechanism provided for an exemplary embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a detailed process flow of mutation point detection provided by an exemplary embodiment of the present invention;

[0043] Figure 6 A schematic diagram of a monitoring indicator adaptive full-process processing system provided by an exemplary embodiment of the present invention;

[0044] Figure 7 A schematic diagram of a computer device is provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0047] As described in the background technology, the existing monitoring indicator metadata processing solutions have the following problems: (1) Collection is rigid and cannot adapt to dynamically changing monitoring indicators. The indicator metadata (such as indicator name, data type, collection method) of the monitoring object (such as database, middleware, operating system, etc.) may change due to version upgrades or configuration changes, but the existing systems (such as Prometheus, Zabbix) rely on manual maintenance of collection templates and cannot automatically perceive changes, resulting in data loss or redundant collection. Operation and maintenance personnel need to frequently manually adjust the collection strategy, which has high maintenance costs and may cause monitoring blind spots due to update delays; (2) Heterogeneous data integration is difficult, the collection method is not unified, and the indicator output formats of different monitoring objects (such as Redis, MySQL, Kafka) vary greatly (such as Prometheus format, JSON, key-value pairs). , the existing system needs to customize the parsing logic for each data source, which has poor scalability. When adding new monitoring objects, it is necessary to develop adaptation code, and the system is difficult to quickly support new components; (3) The contradiction between storage efficiency and exception capture. The traditional storage solution adopts fixed-frequency sampling (such as once per minute) and simple aggregation (such as average value), resulting in data redundancy in the stable period (such as long-term stability of CPU utilization). Transient mutations (such as network instantaneous jitter) are smoothed and ignored due to the sampling interval or aggregation algorithm, resulting in high storage costs, and key exceptions may be missed, affecting fault diagnosis; (4) High-latency acquisition leads to insufficient real-time performance. High-compression storage (such as 10s aggregation window) is prone to losing transient exceptions (such as instantaneous increase in CPU), and high-frequency sampling will lead to storage expansion. Existing algorithms (such as fixed threshold filtering) are difficult to balance the two and cannot meet real-time monitoring needs. The data is incomplete during fault backtracing.

[0048] In view of the problems existing in the current solution, this implementation proposes a monitoring indicator adaptive full-process processing method, which includes the following steps:

[0049] S101: Acquire monitoring indicator metadata of a collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period;

[0050] S102: Normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store the data groups without identified mutation points according to the ratio of storage frequency to acquisition frequency, and compress and store the data groups with identified mutation points according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0051] like Figure 1 As shown, this implementation method provides a specific overall implementation process:

[0052] Start: process starts;

[0053] Raw data: Get raw data (i.e., collected monitoring indicator metadata) as input;

[0054] Memory ring buffer area: the original data is stored in the memory ring buffer area for subsequent processing;

[0055] Data processing window: Extracts monitoring indicator metadata from the memory ring buffer area to form a data processing window (each window represents a data group);

[0056] Determine whether the window data contains a mutation point: If the window data does not contain a mutation point, enter "Generate Statistics" to generate the corresponding statistical value; if the window data contains a mutation point, calculate the statistical value, the mutation point, and the collection value points "before 1 and after 2" of the mutation point;

[0057] Intelligent data aggregation: The statistical values ​​generated in the above steps (regardless of whether they contain mutation points) are intelligently aggregated.

[0058] End: The process ends.

[0059] Before S101, the process of configuring monitoring indicator objects on the visual interface also includes:

[0060] Configure monitoring objects on the interface, including the monitoring indicator object name, type, indicator collection address (optional), whether to collect and other configuration information; for common middleware (such as Redis and MySQL), there is no need to configure the collection address, and indicator metadata is automatically identified; specify key information such as the indicator collection address through the interface, and use a custom collection address to identify metadata; after enabling the collection configuration, push the data to the monitoring indicator metadata processing queue, providing a unified entry point for subsequent analysis and processing.

[0061] The specific process of S101 of the present embodiment includes:

[0062] The subscription metadata processing queue acquires monitoring index metadata of the collection object according to the collection data object type and the index collection address, formats the monitoring index metadata, and records the monitoring index metadata into a file (i.e., a first setting file). The operation of the recording file is performed only once a day (here, the event interval can also be selected according to specific working conditions, for example, it can be selected to be performed once every 12 hours, or once every 2 days, which will not be described here). The monitoring index metadata processing module (which can also be other special processing modules) is notified to synchronize the monitoring index metadata into a relational database.

[0063] In the present embodiment, due to component version upgrade and API modification, etc., the monitoring index metadata can change. The monitoring index metadata is acquired daily and written into a new file (i.e., a second setting file). The index metadata files of the same monitoring object of the new and old files are compared. If the monitoring index metadata does not change, the new file (i.e., the second setting file) is deleted. If there is a difference, the difference metadata is compared. The content of the new file (i.e., the second setting file) covers the old file (i.e., the first setting file). The monitoring index metadata processing module is notified to process the monitoring index metadata. The difference metadata is pushed to the user end. The user confirms whether to replace or add or delete, so as to re-maintain the index metadata in the time series database.

[0064] As shown in FIG. 1, a specific execution process is given, which includes: Figure 2

[0065] Start: the process starts;

[0066] Interface configuration monitoring index object (enabled): configure and enable the index object to be monitored on the interface;

[0067] Automatic discovery / self-defined configuration: select automatic discovery metadata, or parse metadata according to the configured data collection address;

[0068] Data acquisition and writing: automatically acquire metadata and write the metadata into a metadata queue;

[0069] Generate today's data file: generate today's data file according to the metadata queue;

[0070] Determine whether it is the first generation: if it is the first generation, synchronize the data in the data file to the time series database. If it is not the first generation, check whether there is a yesterday's file;

[0071] ​Compare yesterday's file: If yesterday's file does not exist, delete the new file (which may be redundant or incorrectly generated); if yesterday's file exists, compare yesterday's file with today's file;

[0072] Determine whether there are differences: If no changes exist, delete the new file; if differences exist, generate a difference report, notify the user to handle the issue, adjust the display name of the change indicator and other information, and then delete the old file;

[0073] End: The process ends.

[0074] In S102 of this implementation, different monitoring objects have different indicator collection methods (including but not limited to TCP acquisition, API request acquisition, etc.), resulting in different formats of returned indicator data, including object format, Map format, and Prometheus format. Therefore, this implementation uses a data formatting module to normalize the data (converting the original data into a standard time series format).

[0075] like Figure 3 As shown, this implementation provides a specific normalization process, including:

[0076] Start: process starts;

[0077] Raw data: obtain the data to be normalized;

[0078] Format judgment: judge the format of the data to be normalized, and take different processing paths according to different formats, as follows:

[0079] Prometheus / openMetrics format: If the original data is in Prometheus or openMetrics format, perform the "parse metrics text" operation;

[0080] JSON / Map format: If the original data is in JSON or Map format, perform the "Extract key-value pairs" operation;

[0081] TCP / UDP stream format: If the original data is in TCP or UDP stream format, perform the "parse data by protocol" operation;

[0082] Unify data into time series format: The data obtained after processing in different formats mentioned above are uniformly converted into time series format data;

[0083] Push to storage module: Push the time series data in a unified format to the storage module;

[0084] End: The process ends.

[0085] The acquisition frequency of the collector in this implementation is set lower than the storage frequency (can be set according to the ratio of 1:10). At the same time, a thread-safe double buffer setting is adopted. The Active Buffer is used to receive new real-time data, and the Ready Buffer is used to process window data. The use of double buffer setting can effectively avoid data being locked. After the buffer is exchanged according to the storage frequency (the data in the Active Buffer enters the Ready Buffer), the data is processed. The specific process is as follows Figure 4 Shown, including:

[0086] Data writing: The metadata of the new monitoring indicator is written to the Active Buffer.

[0087] Compression condition judgment: Continue writing data to the Active Buffer until the preset compression ratio condition is reached;

[0088] Periodic trigger check: After the compression ratio is reached, the data in the active buffer is checked periodically.

[0089] Mutation data judgment: Check whether there is mutation data in the window data; if there is no mutation data: If there is no mutation data in the window data, perform statistical aggregation operations (for example, by generating statistical values); if there is mutation data: If there is mutation data in the window data, retain the mutation point and its context data;

[0090] Data compression: Regardless of whether statistical aggregation or the retention of mutation points and context data is performed, the processed data is ultimately compressed to obtain compressed data.

[0091] In this implementation, the collected data is analyzed to identify the mutation points. The improved Hampel filter algorithm is used to identify the mutation points through the median absolute deviation. Specifically, the following steps are performed:

[0092] (1);

[0093] in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median;

[0094] When data points When it is greater than the set threshold, the The data points are mutation points. Compared with the traditional method based on mean ± standard deviation (σ), the Hampel filter is more robust to outliers because the median is less affected by extreme values.

[0095] In this implementation, after a mutation point is detected, the first and second adjacent data points are retained and ultimately stored in the time series library, ensuring that valuable data is not discarded. By retaining the first data point, the final normal state before the mutation is provided and the mutation point positioning deviation caused by filtering algorithm delay is avoided. By retaining the second and third data points, the persistence of the mutation and the changing trend of subsequent data can be verified, and the instantaneous reaction after the mutation can be captured. The setting of the first and second data points does not significantly increase the number of retained data points while still achieving the intended function.

[0096] Before writing into the time series database, the collected indicator data is preprocessed.

[0097] (1) Intelligent aggregation, the goal is to reduce the number of write points and retain valuable data. For a set of data that changes steadily, retain , N represents the total number of data points in the data group, Represents the storage frequency, Represents the acquisition frequency, significantly reducing redundant points; for a set of data with mutation values, the number of retained data points is: ,in, Represents the number of mutation points, It means retaining each mutation point as well as the first and last two data points of each mutation point.

[0098] (2) High-reliability writing: writing data into the time series database in batches to reduce network overhead; retaining and rewriting data blocks that failed to be written to ensure zero data loss as much as possible.

[0099] The detailed process of mutation point detection in this implementation is as follows Figure 5 Shown, including:

[0100] Get data: First, get the sliding window data as the basic data set for subsequent analysis;

[0101] Calculate statistics: Calculate the median and mean absolute deviation (MAD) for the sliding window data. These two statistics are used to measure the central tendency and dispersion of the data;

[0102] Marking over-threshold points: Determine a threshold based on the calculated median and MAD, and mark the data points in the sliding window data that exceed the threshold as over-threshold points;

[0103] Judging continuous breakthrough: checking whether the marked threshold-crossing point exists continuous breakthrough: if yes (continuous breakthrough exists, i.e. multiple continuous data points cross the threshold), the data enters the peak preservation state; if no (continuous breakthrough does not exist), the threshold-crossing point is marked as a single point;

[0104] Preserving relevant data points: whether the data enters the peak preservation state or is marked as a single point, the mutation point and its context data points (1 before and 2 after) are preserved;

[0105] Generating an abnormal event: based on the preserved mutation point and context data points, an abnormal event is generated for further analysis and processing.

[0106] Figure 6 A monitoring index adaptive full-process processing system is shown, comprising:

[0107] A data acquisition unit 601 is configured to acquire monitoring index metadata of a collection object, format and store the monitoring index metadata, and update the stored data according to newly collected monitoring index metadata every interval of a set time period;

[0108] A compressed storage unit 602 is configured to perform normalization processing on the updated index metadata, perform group mutation point detection on the normalized index metadata, compress and store data groups without identified mutation points according to the ratio of storage frequency to collection frequency, and compress and store data groups with identified mutation points according to the ratio of storage frequency to collection frequency and a set number of data points before and after the mutation points.

[0109] It can be understood that the above-mentioned units can be combined into one or several other units respectively or all, or some of the units can be further split into a plurality of units with smaller functions to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions, and in actual application, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the system can also include other units, and in actual application, these functions can also be realized by other units, and can be realized by multiple units in cooperation.

[0110] According to another embodiment of the present invention, the system described in this embodiment can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general-purpose computing device such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0111] Figure 7 7 shows a computer device, which includes a processor 701, a communication interface 702, and a computer-readable storage medium 703. The processor 701, the communication interface 702, and the computer-readable storage medium 703 may be connected via a bus or other means.

[0112] Among them, the communication interface 702 is used to receive and send data, the computer-readable storage medium 703 can be stored in the memory of the electronic device, the computer-readable storage medium 703 is used to store computer programs, the computer programs include program instructions, and the processor 701 is used to execute the program instructions stored in the computer-readable storage medium 703.

[0113] The processor 701 is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0114] The processor 701 is configured to perform the following process:

[0115] Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period;

[0116] The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0117] The present invention also provides a computer-readable storage medium, which is a memory device in an electronic device for storing programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides storage space that stores the processing system of the electronic device.

[0118] Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by the processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; alternatively, it may be at least one computer-readable storage medium located remotely from the processor.

[0119] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0120] Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period;

[0121] The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0122] The present invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the following process:

[0123] Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period;

[0124] The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0125] Those skilled in the art will appreciate that the units and algorithmic steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0126] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. A computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they fully or partially produce the processes or functions according to the embodiments of the present invention. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data processing device such as a server or data center that integrates one or more available media. Available media can include magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0127] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A monitoring indicator adaptive full-process processing method, characterized in that: The following processes are included: Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; Normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store data groups without identified mutation points based on the ratio of storage frequency to acquisition frequency, and compress and store data groups with identified mutation points based on the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point; The median absolute deviation is used to identify the mutation points of each data group, including: ； in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median; When data points When it is greater than the set threshold, the The data points are mutation points.

2. The monitoring indicator adaptive full-process processing method according to claim 1, characterized in that: The monitoring indicator metadata is formatted and stored, and the stored data is updated according to the newly collected monitoring indicator metadata after each set period, including: Subscribe to the metadata processing queue, obtain the monitoring indicator metadata of the collection object according to the collection data object type and indicator collection address, format the metadata, and record it in the first setting file; After each set period, the monitoring indicator metadata of the collection object is obtained again and written into a new second setting file. The first setting file and the second setting file are compared. If the monitoring indicator metadata has not changed, the second setting file is deleted. If there is a difference, the difference metadata is extracted and the second setting file is used to overwrite the first setting file. The difference items are sent to the preset client for data change reminders.

3. The monitoring indicator adaptive full-process processing method according to claim 2, characterized in that: After each set period, the monitoring indicator metadata of the collection object is obtained again. The following processes are included: Determine whether the monitoring indicator metadata in the second setting file is generated for the first time: If it is generated for the first time, synchronize the monitoring indicator metadata directly to the time series database; if it is not generated for the first time, compare it with the monitoring indicator metadata in the first setting file.

4. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 3, characterized in that: For the data set where no mutation point is identified, the number of retained data points is: , N represents the total number of data points in the data group, Represents the storage frequency, Represents the acquisition frequency.

5. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 3, characterized in that: For the data set with mutation points identified, the number of retained data points is: , where N represents the total number of data points in the data set, Represents the storage frequency, represents the acquisition frequency, Represents the number of mutation points, It means retaining each mutation point as well as the first and last two data points of each mutation point.

6. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 3, characterized in that: Before obtaining the monitoring indicator metadata of the collection object, The following processes are included: After configuring the monitoring indicator object and enabling the collection configuration, the collected data will be automatically pushed to the monitoring indicator metadata processing queue. The monitoring indicator metadata for processing can be extracted by subscribing to the monitoring indicator metadata processing queue.

7. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 3, characterized in that: A thread double buffer setting is used, the Active Buffer is used to receive new real-time data, and the Ready Buffer is used to process window data.

8. A monitoring indicator adaptive full-process processing system, characterized in that: include: The data acquisition unit is configured to: acquire monitoring indicator metadata of the acquisition object, format the monitoring indicator metadata and store it, and update the stored data according to the newly acquired monitoring indicator metadata after each set period of time; The compression storage unit is configured to: normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store data groups for which no mutation point is identified based on a ratio of storage frequency to acquisition frequency, and compress and store data groups for which mutation point is identified based on a ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point; The median absolute deviation is used to identify the mutation points of each data group, including: ； in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median; When data points When it is greater than the set threshold, the The data points are mutation points.

9. A computer device, characterized in that: include: a processor and a computer-readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the monitoring indicator adaptive full-process processing method according to any one of claims 1 to 7.

Images