Monitoring index self-adaptive full-process processing method and system

Abstract

The invention belongs to the technical field of data processing. The invention provides a monitoring index self-adaptive full-process processing method and system, and the method comprises the steps: obtaining monitoring index metadata of a collection object, carrying out the formatting processing of the monitoring index metadata, storing the formatted monitoring index metadata, and updating the stored data according to the newly collected monitoring index metadata at a set time interval; performing normalization processing on the updated index metadata, performing grouping mutation point detection on the normalized index metadata, performing compression storage on a data group of which a mutation point is not identified according to a ratio of a storage frequency to an acquisition frequency, and performing compression storage on a data group of which the mutation point is identified according to a ratio of a storage frequency to an acquisition frequency. Performing compression storage according to the ratio of the storage frequency to the acquisition frequency and the set number of data points before and after the mutation point; and the storage efficiency and the data fidelity are effectively considered.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a monitoring indicator adaptive full-process processing method and system. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of IoT technology, monitoring systems are facing unprecedented technical challenges and innovative opportunities. With the in-depth application of next-generation information technologies such as 5G and edge computing, modern monitoring requirements have expanded from traditional hardware performance metrics (such as basic resources like CPU, memory, and disk) to comprehensive monitoring systems covering the entire stack, including middleware, databases, and operating systems. Furthermore, key business metrics such as API response latency, message queue depth, and file system capacity must be collected. This diversified evolution of monitoring dimensions has made traditional monitoring solutions based on static configurations inadequate when dealing with dynamically changing metadata, integrating heterogeneous data sources, and storing massive amounts of time-series data.

[0004] The main problems with existing solutions are: delayed response to metadata changes, making it impossible to timely perceive structural changes in monitored objects; rigid data collection methods that are difficult to adapt to the characteristics of different monitored objects; and difficulty balancing storage efficiency and data fidelity, often resulting in a loss of one while focusing on the other. This is especially true in quasi-real-time monitoring scenarios where pull mode is used for data collection (typical sampling intervals range from 10 seconds to several minutes). Although the traditional long storage window design effectively reduces storage overhead, it is prone to losing transient abnormal data within the window (such as sudden jitter, instantaneous traffic peaks, etc.). This data distortion may cause serious operational blind spots and even lead to misdiagnosis of faults. Summary of the Invention

[0005] In order to address the shortcomings of the existing technology, the present invention provides a monitoring indicator adaptive full-process processing method and system, which realizes the full-process automated processing of the collection, processing and storage of monitoring indicator metadata. For data groups without identified mutation points, compressed storage is performed according to the ratio of storage frequency to acquisition frequency. For data groups with identified mutation points, compressed storage is performed according to the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point, which effectively takes into account both storage efficiency and data fidelity, and improves data processing efficiency.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for adaptive full-process processing of monitoring indicators.

[0007] A monitoring indicator adaptive full-process processing method includes the following steps: Obtain the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0008] In one implementation of the first aspect of the present invention, the monitoring indicator metadata is formatted and stored, and after each set period, the stored data is updated based on the newly collected monitoring indicator metadata, including: Subscribe to the metadata processing queue, obtain the monitoring indicator metadata of the collection object according to the collection data object type and indicator collection address, format the metadata, and record it in the first setting file; After each set period, the monitoring indicator metadata of the collection object is obtained again and written into a new second setting file. The first setting file and the second setting file are compared. If the monitoring indicator metadata has not changed, the second setting file is deleted. If there is a difference, the difference metadata is extracted and the second setting file is used to overwrite the first setting file. The difference items are sent to the preset client for data change reminders.

[0009] As a further limitation of the first aspect of the present invention, obtaining the monitoring indicator metadata of the collection object again after each set period of time also includes the following process: Determine whether the monitoring indicator metadata in the second setting file is generated for the first time: If it is generated for the first time, synchronize the monitoring indicator metadata directly to the time series database; if it is not generated for the first time, compare it with the monitoring indicator metadata in the first setting file.

[0010] In an implementation of the first aspect of the present invention, identifying the mutation point of each data group by median absolute deviation includes: ,in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median; When data points When it is greater than the set threshold, the The data points are mutation points.

[0011] In one implementation of the first aspect of the present invention, for a data set in which no mutation point is identified, the number of retained data points is: , N represents the total number of data points in the data group, Represents the storage frequency, Represents the acquisition frequency.

[0012] In one implementation of the first aspect of the present invention, for a data set in which a mutation point is identified, the number of retained data points is: , where N represents the total number of data points in the data set, Represents the storage frequency, represents the acquisition frequency, Represents the number of mutation points, It means retaining each mutation point as well as the first and last two data points of each mutation point.

[0013] In an implementation of the first aspect of the present invention, before obtaining the monitoring indicator metadata of the collection object, the following process is also included: After configuring the monitoring indicator object and enabling the collection configuration, the collected data will be automatically pushed to the monitoring indicator metadata processing queue. The monitoring indicator metadata for processing can be extracted by subscribing to the monitoring indicator metadata processing queue.

[0014] In one implementation of the first aspect of the present invention, a thread double buffering setting is adopted, in which the Active Buffer is used to receive new real-time data, and the Ready Buffer is used to process window data. This can effectively prevent data from being locked. After the buffers are exchanged according to the storage frequency (data in the Active Buffer enters the Ready Buffer), the data is processed.

[0015] In a second aspect, the present invention provides a monitoring indicator adaptive full-process processing system.

[0016] A monitoring indicator adaptive full-process processing system, comprising: The data acquisition unit is configured to: acquire monitoring indicator metadata of the acquisition object, format the monitoring indicator metadata and store it, and update the stored data according to the newly acquired monitoring indicator metadata after each set period; The compression storage unit is configured to: normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store the data group for which no mutation point is identified according to the ratio of storage frequency to acquisition frequency, and compress and store the data group for which mutation point is identified according to the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point.

[0017] In a third aspect, the present invention provides a computer device comprising: a processor and a computer-readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the monitoring indicator adaptive full-process processing method as described in the first aspect of the present invention.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention innovatively proposes an adaptive full-process processing method for monitoring indicators, which realizes the full-process automated processing of the collection, processing and storage of monitoring indicator metadata. For data groups without identified mutation points, compression storage is performed based on the ratio of storage frequency to acquisition frequency. For data groups with identified mutation points, compression storage is performed based on the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point. This effectively takes into account both storage efficiency and data fidelity, and improves the efficiency of data processing.

[0019] 2. The present invention innovatively proposes a full-process adaptive processing method for monitoring indicators. Metadata is formatted and recorded in a first setting file. After each set period, the monitoring indicator metadata of the collection object is obtained again and written to a new second setting file. The first setting file and the second setting file are compared. If the monitoring indicator metadata has not changed, the second setting file is deleted. If there is a difference, the difference metadata is extracted and the second setting file is used to overwrite the first setting file. The difference items are sent to the preset client as a data change reminder. Changes in the monitoring indicator of the monitoring object (such as the addition / deletion of indicators) are automatically detected without manual maintenance of the collection template. Through the difference comparison and user confirmation process, the accuracy and flexibility of metadata updates are ensured, and monitoring interruptions caused by version upgrades are avoided.

[0020] 3. The present invention innovatively proposes an adaptive full-process processing method for monitoring indicators, normalizes the updated indicator metadata, supports multiple heterogeneous data formats such as Prometheus, JSON, key-value pairs, etc., and only requires configuration when adding new monitoring objects without the need to develop adaptation code. It supports multiple collection methods such as API pull, TCP, UDP monitoring, and adapts to the complex environment of the Internet of Things.

[0021] 4. This invention innovatively proposes an adaptive full-process processing method for monitoring indicators. For stable data groups, it downsamples proportionally to reduce redundant storage. For data groups with mutation points, it accurately identifies anomalies through an improved Hampel filtering algorithm and expands the storage context (retaining the "first 1 and last 2" data points). By combining block submission of data with breakpoint retry, it ensures low data loss.

[0022] 5. The present invention innovatively proposes an adaptive full-process processing method for monitoring indicators, which adopts double-buffered high-frequency sampling. The bottom layer collects data at a high frequency (such as 1 second / time), and the upper layer downsamples and stores it on demand, which not only avoids missing transient anomalies but also controls storage expansion. At the same time, the storage granularity is dynamically adjusted based on data fluctuations, with high fidelity for key data and high compression for non-key data, thereby improving processing efficiency.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 A schematic diagram of a data processing flow provided for an exemplary embodiment of the present invention; Figure 2 A schematic diagram of a process for monitoring indicator metadata management provided by an exemplary embodiment of the present invention; Figure 3 A schematic diagram of normalization processing of monitoring indicator data provided by an exemplary embodiment of the present invention; Figure 4 A schematic diagram of a double buffer processing mechanism provided for an exemplary embodiment of the present invention; Figure 5 A schematic diagram of a detailed process flow of mutation point detection provided by an exemplary embodiment of the present invention; Figure 6 A schematic diagram of a monitoring indicator adaptive full-process processing system provided by an exemplary embodiment of the present invention; Figure 7 A schematic diagram of a computer device is provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0028] As described in the background technology, the existing monitoring indicator metadata processing solutions have the following problems: (1) Collection is rigid and cannot adapt to dynamically changing monitoring indicators. The indicator metadata (such as indicator name, data type, collection method) of the monitoring object (such as database, middleware, operating system, etc.) may change due to version upgrades or configuration changes, but the existing systems (such as Prometheus, Zabbix) rely on manual maintenance of collection templates and cannot automatically perceive changes, resulting in data loss or redundant collection. Operation and maintenance personnel need to frequently manually adjust the collection strategy, which has high maintenance costs and may cause monitoring blind spots due to update delays; (2) Heterogeneous data integration is difficult, the collection method is not unified, and the indicator output formats of different monitoring objects (such as Redis, MySQL, Kafka) vary greatly (such as Prometheus format, JSON, key-value pairs). , the existing system needs to customize the parsing logic for each data source, which has poor scalability. When adding new monitoring objects, it is necessary to develop adaptation code, and the system is difficult to quickly support new components; (3) The contradiction between storage efficiency and exception capture. The traditional storage solution adopts fixed-frequency sampling (such as once per minute) and simple aggregation (such as average value), resulting in data redundancy in the stable period (such as long-term stability of CPU utilization). Transient mutations (such as network instantaneous jitter) are smoothed and ignored due to the sampling interval or aggregation algorithm, resulting in high storage costs, and key exceptions may be missed, affecting fault diagnosis; (4) High-latency acquisition leads to insufficient real-time performance. High-compression storage (such as 10s aggregation window) is prone to losing transient exceptions (such as instantaneous increase in CPU), and high-frequency sampling will lead to storage expansion. Existing algorithms (such as fixed threshold filtering) are difficult to balance the two and cannot meet real-time monitoring needs. The data is incomplete during fault backtracing.

[0029] In view of the problems existing in the current solution, this implementation proposes a monitoring indicator adaptive full-process processing method, which includes the following steps: S101: Acquire monitoring indicator metadata of a collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; S102: Normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store the data groups without identified mutation points according to the ratio of storage frequency to acquisition frequency, and compress and store the data groups with identified mutation points according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0030] like Figure 1 As shown, this implementation method provides a specific overall implementation process: Start: process starts; Raw data: Get raw data (i.e., collected monitoring indicator metadata) as input; Memory ring buffer area: the original data is stored in the memory ring buffer area for subsequent processing; Data processing window: Extracts monitoring indicator metadata from the memory ring buffer area to form a data processing window (each window represents a data group); Determine whether the window data contains a mutation point: If the window data does not contain a mutation point, enter "Generate Statistics" to generate the corresponding statistical value; if the window data contains a mutation point, calculate the statistical value, the mutation point, and the collection value points "before 1 and after 2" of the mutation point; Intelligent data aggregation: The statistical values generated in the above steps (regardless of whether they contain mutation points) are intelligently aggregated. End: The process ends.

[0031] Before S101, the process of configuring monitoring indicator objects on the visual interface also includes: Configure monitoring objects on the interface, including the monitoring indicator object name, type, indicator collection address (optional), whether to collect and other configuration information; for common middleware (such as Redis and MySQL), there is no need to configure the collection address, and indicator metadata is automatically identified; specify key information such as the indicator collection address through the interface, and use a custom collection address to identify metadata; after enabling the collection configuration, push the data to the monitoring indicator metadata processing queue, providing a unified entry point for subsequent analysis and processing.

[0032] The specific process of S101 of this implementation includes: Subscribe to the metadata processing queue, obtain the monitoring indicator metadata of the collection object according to the collection data object type and indicator collection address, format the monitoring indicator metadata, and record it in a file (i.e., the first setting file). The file recording operation is only performed once a day (the event interval here can also be selected according to the specific working conditions, for example, once every 12 hours or once every 2 days, which will not be repeated here), and notify the indicator metadata processing module (or other dedicated processing modules) to synchronize the monitoring indicator metadata to the relational database.

[0033] In this implementation, due to issues such as component version upgrades and API modifications, the monitoring indicator metadata may change. The monitoring indicator metadata is obtained on a daily basis and written to a new file (i.e., the second setting file). The new and old indicator metadata files of the same monitoring object are compared. If the monitoring indicator metadata has not changed, the new file (i.e., the second setting file) is deleted. If there is a difference, the difference metadata is compared, and the new file (i.e., the second setting file) overwrites the old file (i.e., the first setting file). At the same time, the indicator metadata processing module is notified to process the monitoring indicator metadata, and the difference metadata is pushed to the user end. The user confirms whether to replace or add or delete, thereby re-maintaining the indicator metadata in the time series database.

[0034] like Figure 2 As shown in FIG, a specific execution process is given, including: Start: process starts; Interface configuration monitoring indicator object (enable): configure and enable the indicator object to be monitored on the interface; Automatic discovery / custom configuration: Choose to automatically discover metadata, or collect address resolution metadata based on the configured data; Data acquisition and writing: automatically acquire metadata and write the metadata into the metadata queue; Generate the data file of the day: Generate the data file of the day according to the metadata queue; Determine whether it is the first generation: If it is the first generation, synchronize the data in the data file to the time series database. If it is not the first generation, check whether there is yesterday's file. Compare yesterday's file: If yesterday's file does not exist, delete the new file (which may be redundant or incorrectly generated); if yesterday's file exists, compare yesterday's file with today's file; Determine whether there are differences: If no changes exist, delete the new file; if differences exist, generate a difference report, notify the user to handle the issue, adjust the display name of the change indicator and other information, and then delete the old file; End: The process ends.

[0035] In S102 of this implementation, different monitoring objects have different indicator collection methods (including but not limited to TCP acquisition, API request acquisition, etc.), resulting in different formats of returned indicator data, including object format, Map format, and Prometheus format. Therefore, this implementation uses a data formatting module to normalize the data (converting the original data into a standard time series format).

[0036] like Figure 3 As shown, this implementation provides a specific normalization process, including: Start: process starts; Raw data: obtain the data to be normalized; Format judgment: judge the format of the data to be normalized, and take different processing paths according to different formats, as follows: Prometheus / openMetrics format: If the original data is in Prometheus or openMetrics format, perform the "parse metrics text" operation; JSON / Map format: If the original data is in JSON or Map format, perform the "Extract key-value pairs" operation; TCP / UDP stream format: If the original data is in TCP or UDP stream format, perform the "parse data by protocol" operation; Unify data into time series format: The data obtained after processing in different formats mentioned above are uniformly converted into time series format data; Push to storage module: Push the time series data in a unified format to the storage module; End: The process ends.

[0037] The acquisition frequency of the collector in this implementation is set lower than the storage frequency (can be set according to the ratio of 1:10). At the same time, a thread-safe double buffer setting is adopted. The Active Buffer is used to receive new real-time data, and the Ready Buffer is used to process window data. The use of double buffer setting can effectively avoid data being locked. After the buffer is exchanged according to the storage frequency (the data in the Active Buffer enters the Ready Buffer), the data is processed. The specific process is as follows Figure 4 Shown, including: Data writing: The metadata of the new monitoring indicator is written to the Active Buffer. Compression condition judgment: Continue writing data to the Active Buffer until the preset compression ratio condition is reached; Periodic trigger check: After the compression ratio is reached, the data in the active buffer is checked periodically. Mutation data judgment: Check whether there is mutation data in the window data; if there is no mutation data: If there is no mutation data in the window data, perform statistical aggregation operations (for example, by generating statistical values); if there is mutation data: If there is mutation data in the window data, retain the mutation point and its context data; Data compression: Regardless of whether statistical aggregation or the retention of mutation points and context data is performed, the processed data is ultimately compressed to obtain compressed data.

[0038] In this implementation, the collected data is analyzed to identify the mutation points. The improved Hampel filter algorithm is used to identify the mutation points through the median absolute deviation. Specifically, the following steps are performed: (1); in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median; When data points When it is greater than the set threshold, the The data points are mutation points. Compared with the traditional method based on mean ± standard deviation (σ), the Hampel filter is more robust to outliers because the median is less affected by extreme values.

[0039] In this implementation, after a mutation point is detected, the first and second adjacent data points are retained and ultimately stored in the time series library, ensuring that valuable data is not discarded. By retaining the first data point, the final normal state before the mutation is provided and the mutation point positioning deviation caused by filtering algorithm delay is avoided. By retaining the second and third data points, the persistence of the mutation and the changing trend of subsequent data can be verified, and the instantaneous reaction after the mutation can be captured. The setting of the first and second data points does not significantly increase the number of retained data points while still achieving the intended function.

[0040] Before writing into the time series database, the collected indicator data is preprocessed.

[0041] (1) Intelligent aggregation, the goal is to reduce the number of write points and retain valuable data. For a set of data that changes steadily, retain , N represents the total number of data points in the data group, Represents the storage frequency, Represents the acquisition frequency, significantly reducing redundant points; for a set of data with mutation values, the number of retained data points is: ,in, Represents the number of mutation points, It means retaining each mutation point as well as the first and last two data points of each mutation point.

[0042] (2) High-reliability writing: writing data into the time series database in batches to reduce network overhead; retaining and rewriting data blocks that failed to be written to ensure zero data loss as much as possible.

[0043] The detailed process of mutation point detection in this implementation is as follows Figure 5 Shown, including: Get data: First, get the sliding window data as the basic data set for subsequent analysis; Calculate statistics: Calculate the median and mean absolute deviation (MAD) for the sliding window data. These two statistics are used to measure the central tendency and dispersion of the data; Marking over-threshold points: Determine a threshold based on the calculated median and MAD, and mark the data points in the sliding window data that exceed the threshold as over-threshold points; Determine the continuous breakthrough situation: Check whether the marked super-threshold points have continuous breakthroughs: If so (continuous breakthroughs exist, that is, multiple consecutive data points exceed the threshold), the data enters the peak preservation state; if not (no continuous breakthroughs exist), the super-threshold points are marked as single points; Retain relevant data points. Regardless of whether the data enters the peak preservation state or is single-point labeled, the mutation point and its context data points (1 before and 2 after) are retained. Generate abnormal events: Generate abnormal events based on the retained mutation points and contextual data points for further analysis and processing.

[0044] Figure 6 A monitoring indicator adaptive full-process processing system is shown, including: The data acquisition unit 601 is configured to: acquire monitoring indicator metadata of the acquisition object, format the monitoring indicator metadata and store it, and update the stored data according to the newly acquired monitoring indicator metadata after each set period; The compression storage unit 602 is configured to: normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store the data group for which no mutation point is identified based on the ratio of storage frequency to acquisition frequency, and compress and store the data group for which mutation point is identified based on the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point.

[0045] It is understandable that each of the above-mentioned units can be separately or completely combined into one or several other units to form a whole, or one (or more) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present invention, the system can also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0046] According to another embodiment of the present invention, the system described in this embodiment can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general-purpose computing device such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0047] Figure 7 7 shows a computer device, which includes a processor 701, a communication interface 702, and a computer-readable storage medium 703. The processor 701, the communication interface 702, and the computer-readable storage medium 703 may be connected via a bus or other means.

[0048] Among them, the communication interface 702 is used to receive and send data, the computer-readable storage medium 703 can be stored in the memory of the electronic device, the computer-readable storage medium 703 is used to store computer programs, the computer programs include program instructions, and the processor 701 is used to execute the program instructions stored in the computer-readable storage medium 703.

[0049] The processor 701 is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0050] The processor 701 is configured to perform the following process: Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0051] The present invention also provides a computer-readable storage medium, which is a memory device in an electronic device for storing programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides storage space that stores the processing system of the electronic device.

[0052] Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by the processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; alternatively, it may be at least one computer-readable storage medium located remotely from the processor.

[0053] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process: Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0054] The present invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the following process: Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

[0055] Those skilled in the art will appreciate that the units and algorithmic steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0056] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. A computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they fully or partially produce the processes or functions according to the embodiments of the present invention. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data processing device such as a server or data center that integrates one or more available media. Available media can include magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0057] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A monitoring indicator adaptive full-process processing method, characterized in that: The following processes are included: Acquire the monitoring indicator metadata of the collection object, format the monitoring indicator metadata and store it, and update the stored data according to the newly collected monitoring indicator metadata after each set period; The updated indicator metadata is normalized, and group mutation point detection is performed on the normalized indicator metadata. For data groups where no mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency. For data groups where mutation points are identified, they are compressed and stored according to the ratio of storage frequency to acquisition frequency and the set number of data points before and after the mutation point.

2. The monitoring indicator adaptive full-process processing method according to claim 1, characterized in that: The monitoring indicator metadata is formatted and stored, and the stored data is updated according to the newly collected monitoring indicator metadata after each set period, including: Subscribe to the metadata processing queue, obtain the monitoring indicator metadata of the collection object according to the collection data object type and indicator collection address, format the metadata, and record it in the first setting file; After each set period, the monitoring indicator metadata of the collection object is obtained again and written into a new second setting file. The first setting file and the second setting file are compared. If the monitoring indicator metadata has not changed, the second setting file is deleted. If there is a difference, the difference metadata is extracted and the second setting file is used to overwrite the first setting file. The difference items are sent to the preset client for data change reminders.

3. The monitoring indicator adaptive full-process processing method according to claim 2, characterized in that: After each set period, the monitoring indicator metadata of the collection object is obtained again. The following processes are included: Determine whether the monitoring indicator metadata in the second setting file is generated for the first time: If it is generated for the first time, synchronize the monitoring indicator metadata directly to the time series database; if it is not generated for the first time, compare it with the monitoring indicator metadata in the first setting file.

4. The monitoring indicator adaptive full-process processing method according to claim 1, characterized in that: The median absolute deviation is used to identify the mutation points of each data group, including: ； in, represents the median absolute deviation, Represents the median of the data sequence X in a certain data group. is the data sequence, Representing data series The data points, Representative The absolute deviation of a data point from the median; When data points When it is greater than the set threshold, the The data points are mutation points.

5. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 4, characterized in that: For the data set where no mutation point is identified, the number of retained data points is: , N represents the total number of data points in the data group, Represents the storage frequency, Represents the acquisition frequency.

6. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 4, characterized in that: For the data set with mutation points identified, the number of retained data points is: , where N represents the total number of data points in the data set, Represents the storage frequency, represents the acquisition frequency, Represents the number of mutation points, It means retaining each mutation point as well as the first and last two data points of each mutation point.

7. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 4, characterized in that: Before obtaining the monitoring indicator metadata of the collection object, The following processes are included: After configuring the monitoring indicator object and enabling the collection configuration, the collected data will be automatically pushed to the monitoring indicator metadata processing queue. The monitoring indicator metadata for processing can be extracted by subscribing to the monitoring indicator metadata processing queue.

8. The monitoring indicator adaptive full-process processing method according to any one of claims 1 to 4, characterized in that: A thread double buffer setting is used, the Active Buffer is used to receive new real-time data, and the Ready Buffer is used to process window data.

9. A monitoring indicator adaptive full-process processing system, characterized in that: include: The data acquisition unit is configured to: acquire monitoring indicator metadata of the acquisition object, format the monitoring indicator metadata and store it, and update the stored data according to the newly acquired monitoring indicator metadata after each set period of time; The compression storage unit is configured to: normalize the updated indicator metadata, perform group mutation point detection on the normalized indicator metadata, compress and store the data group for which no mutation point is identified according to the ratio of storage frequency to acquisition frequency, and compress and store the data group for which mutation point is identified according to the ratio of storage frequency to acquisition frequency and a set number of data points before and after the mutation point.

10. A computer device, characterized in that: include: a processor and a computer-readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the monitoring indicator adaptive full-process processing method according to any one of claims 1 to 8 is implemented.

Images