Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of the Internet of Things, and the method comprises the steps: receiving an index event asynchronously published by a target application program through an event bus, determining an event type corresponding to the index event, and routing the index event to a target storage system; obtaining target index data corresponding to the index event through a target storage system, and storing the target index data; the monitoring rear end receives the target index data pushed by the target storage system and displays the target index data, so that loose coupling of an application program and an index collection mechanism is achieved, the flexibility of index collection is improved, meanwhile, the expandability of the mechanism can be achieved by introducing an event bus and index storage, and the reliability of index collection is improved. Batched indexes can be processed at the same time, different requirements are met, and real-time performance is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a data processing method, a data processing device, an electronic device and a computer-readable storage medium. Background Art

[0002] In modern software systems, collecting and monitoring metrics is essential to understanding system performance and health. Monitoring metrics can help us identify performance bottlenecks, detect anomalies, and ensure stable system operation. Usually, monitoring metrics need to be aggregated into a management system for visualization and analysis. However, in the process of collecting and processing system metrics, there are problems with strong coupling, poor scalability, customizability, and real-time performance. Summary of the invention

[0003] The embodiments of the present invention provide a data processing method, device, electronic device and computer-readable storage medium to solve or partially solve the problems of strong coupling, poor scalability, customizability and real-time performance in the process of collecting and processing system indicators.

[0004] An embodiment of the present invention discloses a data processing method, which is applied to a monitoring system, wherein the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system, and the method includes:

[0005] Receiving an index event asynchronously published by a target application through the event bus, determining an event type corresponding to the index event, and routing the index event to a target storage system, wherein the index event is an event obtained by encapsulating target index data generated during the execution of the target application;

[0006] Acquiring target indicator data corresponding to the indicator event through the target storage system, and storing the target indicator data;

[0007] The target indicator data pushed by the target storage system is received through the monitoring backend, and the target indicator data is displayed.

[0008] In some feasible implementations, routing the indicator event to a target storage system includes:

[0009] Obtaining the event type or event content corresponding to the indicator event;

[0010] Determine a target storage system corresponding to the event type or the event content;

[0011] The indicator event is routed to the target storage system.

[0012] In some feasible implementations, the target indicator data includes an indicator name and an indicator value corresponding to the indicator name, and storing the target indicator data includes:

[0013] If the monitoring system runs in a single-threaded environment, determining a first non-thread-safe hash table, and storing the indicator name and the indicator value in the first hash table in the form of a key-value pair;

[0014] If the monitoring system runs in a multi-threaded environment, a thread-safe second hash table is determined, and the indicator name and the indicator value are stored in the second hash table in the form of key-value pairs.

[0015] In some feasible implementations, the storing the target indicator data further includes:

[0016] If the indicator event is a counter type event, then in response to a newly detected counter type event, the target indicator name corresponding to the counter type and the target indicator value corresponding to the target indicator name are determined, and the target indicator value is updated in the first hash table or the second hash table.

[0017] In some feasible implementations, the storage system further includes an indicator refresher located in the storage system, and after storing the target indicator data, the method further includes:

[0018] The indicator refresher responds to the indicator refresh instruction and reads the target indicator data from the storage system according to a preset refresh interval, and batch-packs the target indicator data into a data request;

[0019] The indicator refresher calls the interface or transmission protocol provided by the monitoring backend to push the data request to the monitoring backend.

[0020] Some possible implementations also include:

[0021] The indicator refresher responds to transmitting the indicator data to the monitoring backend. If a network connection abnormality occurs or the monitoring backend is unavailable during the transmission process, the indicator data being transmitted is cached until the network is restored or the monitoring backend is available, and the indicator data is then transmitted to the monitoring backend again.

[0022] Some possible implementations also include:

[0023] If the indicator refresher fails to execute the refresh operation corresponding to the indicator refresh instruction for the first time, the indicator refresher executes the refresh operation again according to the first preset retry interval;

[0024] After each failed retry, the retry interval increases exponentially until the refresh operation is successfully performed.

[0025] In some feasible implementations, after storing the target indicator data, the method further includes:

[0026] If the indicator refresher detects that the amount of indicator data stored in the storage system reaches a preset threshold, or satisfies a preset condition, the target indicator data is read from the storage system, and the target indicator data is batch-packaged into a data request;

[0027] The indicator refresher calls the interface or transmission protocol provided by the monitoring backend to push the data request to the monitoring backend.

[0028] The embodiment of the present invention further discloses a data processing device, which is applied to a monitoring system, wherein the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system, wherein the device includes:

[0029] An event receiving module is used to receive the indicator event asynchronously published by the target application through the event bus, determine the event type corresponding to the indicator event, and route the indicator event to the target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution of the target application;

[0030] A storage module, used to obtain target indicator data corresponding to the indicator event through the target storage system, and store the target indicator data;

[0031] A visualization module is used to receive the target indicator data pushed by the target storage system through the monitoring backend and display the target indicator data.

[0032] In some feasible implementations, the event receiving module is specifically used to:

[0033] Obtaining the event type or event content corresponding to the indicator event;

[0034] Determine a target storage system corresponding to the event type or the event content;

[0035] The indicator event is routed to the target storage system.

[0036] In some feasible implementations, the target indicator data includes an indicator name and an indicator value corresponding to the indicator name, and the storage module is specifically used to:

[0037] If the monitoring system runs in a single-threaded environment, determining a first non-thread-safe hash table, and storing the indicator name and the indicator value in the first hash table in the form of a key-value pair;

[0038] If the monitoring system runs in a multi-threaded environment, a thread-safe second hash table is determined, and the indicator name and the indicator value are stored in the second hash table in the form of key-value pairs.

[0039] In some feasible implementations, the storage module is further used to:

[0040] If the indicator event is a counter type event, then in response to a newly detected counter type event, the target indicator name corresponding to the counter type and the target indicator value corresponding to the target indicator name are determined, and the target indicator value is updated in the first hash table or the second hash table.

[0041] In some feasible implementations, the storage system further includes an indicator refresher located in the storage system, and the device further includes:

[0042] A refresh module, configured to respond to an indicator refresh instruction through the indicator refresher, read the target indicator data from the storage system according to a preset refresh interval, and batch-pack the target indicator data into a data request;

[0043] The push module is used to call the interface or transmission protocol provided by the monitoring backend through the indicator refresher to push the data request to the monitoring backend.

[0044] Some possible implementations also include:

[0045] A cache module is used to respond to the transmission of indicator data to the monitoring backend through the indicator refresher. If a network connection abnormality occurs or the monitoring backend is unavailable during the transmission process, the indicator data being transmitted is cached until the network is restored or the monitoring backend is available, and the indicator data is re-transmitted to the monitoring backend.

[0046] Some possible implementations also include:

[0047] A retry module is used to execute the refresh operation again according to a first preset retry interval if the indicator refresher fails to execute the refresh operation corresponding to the indicator refresh instruction for the first time, wherein after each retry failure, the duration corresponding to the retry interval increases exponentially until the refresh operation is successfully executed.

[0048] In some feasible implementations, the device further includes:

[0049] A response module, configured to read the target indicator data from the storage system and batch-pack the target indicator data into a data request if the indicator refresher detects that the amount of indicator data stored in the storage system reaches a preset threshold or satisfies a preset condition;

[0050] The sending module is used to call the interface or transmission protocol provided by the monitoring backend through the indicator refresher to push the data request to the monitoring backend.

[0051] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0052] The memory is used to store computer programs;

[0053] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0054] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0055] The embodiments of the present invention include the following advantages:

[0056] In an embodiment of the present invention, the monitoring system is applied to the monitoring system, which includes at least an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system. In the process of monitoring the indicators, the system can receive indicator events asynchronously published by the target application through the event bus, determine the event type corresponding to the indicator event, and route the indicator event to the target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution process of the target application. Then, the target indicator data corresponding to the indicator event is obtained through the target storage system, and the target indicator data is stored. Then, the target indicator data pushed by the target storage system is received through the monitoring backend, and the target indicator data is displayed. Therefore, in the process of monitoring the application, the event-based model collects indicators, realizes loose coupling between the application and the indicator collection mechanism, and improves the flexibility of indicator collection. At the same time, the scalability of the mechanism can be achieved by introducing the event bus and indicator storage, and batch indicators can be processed at the same time to meet different needs and ensure real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flowchart of a method for processing data provided in an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of the processing flow of indicator data provided in an embodiment of the present invention;

[0059] Figure 3 It is a structural block diagram of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] As an example, monitoring indicators can help us identify performance bottlenecks, detect anomalies, and ensure stable system operation. Usually, monitoring indicators need to be aggregated into a management system for visualization and analysis. However, in the process of collecting and processing system indicators, there are problems such as strong coupling, poor scalability, customizability, and real-time performance.

[0062] In this regard, in the present invention, it is applied to a monitoring system, and the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system. In the process of monitoring the indicators, the system can receive indicator events asynchronously published by the target application through the event bus, and determine the event type corresponding to the indicator event, and route the indicator event to the target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution process of the target application, and then the target indicator data corresponding to the indicator event is obtained through the target storage system, and the target indicator data is stored, and then the target indicator data pushed by the target storage system is received through the monitoring backend, and the target indicator data is displayed. Therefore, in the process of monitoring the application, the event-based model collects indicators, realizes loose coupling between the application and the indicator collection mechanism, and improves the flexibility of indicator collection. At the same time, the scalability of the mechanism can be achieved by introducing the event bus and indicator storage, and batch indicators can be processed at the same time to meet different needs and ensure real-time performance.

[0063] Reference Figure 1 , shows a flowchart of the steps of a data processing method provided in an embodiment of the present invention, which is applied to a monitoring system, wherein the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system, and specifically may include the following steps:

[0064] Step 101, receiving an index event asynchronously published by a target application through the event bus, determining an event type corresponding to the index event, and routing the index event to a target storage system, wherein the index event is an event obtained by encapsulating target index data generated during the execution of the target application;

[0065] For indicators, it can be used to measure the performance or behavior of the system; for events, it can be a specific action or event that occurs in the application; for event bus, it can be used as a mechanism for transmitting events between different components; for memory storage, it can be used as a mechanism for temporarily storing collected indicator data; for monitoring backend, it can be used as a system for storing and visualizing collected indicators.

[0066] The monitoring system may include an event bus, at least one storage system that is communicatively connected to the event bus, and a monitoring backend that is communicatively connected to the storage system. The event bus may be responsible for receiving events published by the application and routing them to a suitable storage system. The event bus may serve as a middleware, decoupling the application from the storage backend, and enhancing the flexibility and maintainability of the system. The storage system may be used to store the corresponding indicator data. The monitoring backend may be used to visualize the indicator data, so as to monitor the indicator data in real time, and trigger corresponding prompts when the indicator data is abnormal.

[0067] In an embodiment of the present invention, the indicator data involved in the execution of the application is collected and published in an event-driven manner, avoiding the active pulling method of the monitoring backend. Compared with direct indicator collection, the event-based method can provide greater flexibility and scalability.

[0068] In the specific implementation, during the execution of the application, various corresponding indicator data can be generated, such as counters, gauges or histograms, etc. The application can encapsulate these indicators into event objects and publish these indicator events through the event bus. The event bus can receive indicator events asynchronously published by the target application, determine the event type corresponding to the indicator event, and route the indicator event to the target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution process by the target application. The application publishes these indicator events through the event bus, so that the application and the monitoring system can maintain a loose coupling state, and the application only needs to focus on business logic without dealing with the underlying monitoring details.

[0069] It should be noted that the indicator events can be sent asynchronously to avoid blocking the main thread, thereby improving system performance. At the same time, the event model also supports adding new indicator types or expanding different monitoring backends in the future without modifying the existing business logic code.

[0070] Furthermore, the event bus can monitor events in real time. When the target application publishes an indicator event, the event bus automatically captures and processes these events. Specifically, after capturing the indicator event, the event bus can obtain the event type or event content corresponding to the indicator event, then determine the target storage system corresponding to the event type or event content, and then route the indicator event to the target storage system.

[0071] Specifically, the event bus can route the event to the corresponding storage system based on the event type and content. For example, if the event is a counter type, the event bus will route it to a storage module specifically used to store counters; if the event is a measurement value, it will be routed to a measurement value storage module.

[0072] It should be noted that the event bus can allow different types of indicators to be classified and processed, and different indicator types can use different storage and refresh strategies. At the same time, if the system needs to support multiple monitoring backends, the event bus can distribute different events to different storage systems or monitoring backends through routing logic. Then it is routed to the measurement value storage module.

[0073] Step 102, obtaining target indicator data corresponding to the indicator event through the target storage system, and storing the target indicator data;

[0074] When the target storage system receives the indicator event sent by the event bus, it can first obtain the target indicator data corresponding to the indicator event, and then store the target indicator data. The obtained indicator data can be temporarily stored in the memory first, so that a large amount of indicator data can be efficiently saved in a short time, and refreshed to the monitoring backend in batches at an appropriate time. Optionally, the storage system can be similar to a cache to ensure that a large amount of indicator data can be quickly processed in a high-concurrency environment.

[0075] Among them, the target indicator data includes the indicator name and the indicator value corresponding to the indicator name. During the data storage process, if the monitoring system runs in a single-threaded environment, a non-thread-safe first hash table is determined, and the indicator name and the indicator value are stored in the first hash table as a key-value pair; if the monitoring system runs in a multi-threaded environment, a thread-safe second hash table is determined, and the indicator name and the indicator value are stored in the second hash table as a key-value pair, so that the indicator data is stored through different storage methods so that it can be quickly read and updated to ensure the stability of data storage.

[0076] In addition, since the indicator data in the storage system may be dynamically updated, for example, for counter-type indicators, the count value will be updated each time a new event arrives, and the original data will not be overwritten. Therefore, if the indicator event belongs to a counter-type event, then in response to the newly detected counter-type event, the target indicator name corresponding to the counter type and the target indicator value corresponding to the target indicator name are determined, and the target indicator value is updated in the first hash table or the second hash table. Through data updating, it can be ensured that the latest statistical results are stored in the storage system.

[0077] In some feasible implementations, since the storage system is equivalent to a cache, the indicator data cannot be stored in the storage system for a long time. Therefore, in the present application, a corresponding indicator refresher is configured in the storage system. Through the indicator refresher, the indicator data temporarily stored in the storage system can be pushed to the monitoring backend regularly or according to corresponding conditions. On the one hand, it ensures that the indicator data is pushed to the monitoring backend in a timely manner, and on the other hand, it avoids storing too much temporary data in the storage system, which reduces the stability of the system operation.

[0078] After storing the target indicator data, the system can respond to the indicator refresh instruction through the indicator refresher, read the target indicator data from the storage system according to the preset refresh interval, and batch package the target indicator data into data requests, and then call the interface or transmission protocol provided by the monitoring backend through the indicator refresher to push the data request to the monitoring backend.

[0079] Among them, for the indicator refresher, it can use scheduled tasks or schedulers to control the frequency of indicator refresh. Users can set the refresh interval to every 1 minute, 5 minutes or a custom time interval according to actual needs to meet the needs of different scenarios. For example, in high-concurrency scenarios, the refresh frequency can be increased, and in scenarios with limited resources or low real-time requirements, the refresh interval can be appropriately extended.

[0080] It should be noted that in the process of pushing indicator data to the monitoring backend through the indicator refresher, in order to ensure the integrity of the data, when a transmission anomaly occurs, it can be processed through the corresponding processing mechanism (caching mechanism and failure retry mechanism, etc.). Specifically, the indicator refresher can respond to the transmission of indicator data to the monitoring backend. If a network connection anomaly occurs or the monitoring backend is unavailable during the transmission process, the indicator data being transmitted will be cached until the network is restored or the monitoring backend is available, and the indicator data will be re-transmitted to the monitoring backend. In addition, if the indicator refresher fails to execute the refresh operation corresponding to the indicator refresh instruction for the first time, the refresh operation will be executed again according to the first preset retry interval, wherein after each retry failure, the duration corresponding to the retry interval increases exponentially until the refresh operation is successfully executed.

[0081] In addition to the above-mentioned regular push, users can also set corresponding trigger conditions. When the indicator data stored in the storage system meets the trigger conditions, the indicator data push is automatically triggered. Specifically, if the indicator refresher detects that the amount of indicator data stored in the storage system reaches the preset threshold or meets the preset conditions, the target indicator data is read from the storage system, and the target indicator data is batch packaged into a data request. Then, the indicator refresher calls the interface or transmission protocol provided by the monitoring backend to push the data request to the monitoring backend.

[0082] For example, event-driven refresh: when the indicator data reaches a certain number or is triggered by a certain condition, it can be refreshed immediately to ensure that important indicators can be uploaded in time; load-aware refresh: dynamically adjust the refresh frequency according to the current load or performance of the system, such as reducing the refresh frequency under high load to reduce system pressure, etc. The present invention is not limited to this.

[0083] Step 103: receiving the target indicator data pushed by the target storage system through the monitoring backend, and displaying the target indicator data.

[0084] When the indicator refresher pushes the indicator data to the monitoring backend, the monitoring backend can persist the received indicator data and present the corresponding indicator data through the corresponding graphical user interface, so that users can intuitively and quickly understand the corresponding performance conditions through the presented indicator data, so as to make corresponding processing strategies. Therefore, in the process of monitoring the application, the event-based model collects indicators, realizes the loose coupling of the application and the indicator collection mechanism, and improves the flexibility of indicator collection. At the same time, the scalability of the mechanism can be achieved by introducing event bus and indicator storage, and batch indicators can be processed at the same time to meet different needs and ensure real-time performance.

[0085] In one example, suppose there is a microservice responsible for processing user login requests. After each request is processed, the microservice collects indicator data such as request processing time and error rate, and encapsulates this data into an event and publishes it to the event bus. The event bus receives the login request processing event published by the microservice. According to the event type and configuration, the event bus routes the event to the indicator storage component. The indicator storage component receives the login request processing event and extracts indicator data such as request processing time and error rate from it. These data are stored in the ConcurrentHashMap in memory for fast access and processing. Then, the indicator refresher reads the indicator data of login request processing from the memory every minute, and encapsulates this data into a request and sends it to the monitoring backend. After receiving the request, the monitoring backend stores the indicator data in the time series database. If the refresh operation fails, the indicator refresher will retry using an exponential backoff strategy, and log and notify the operation and maintenance personnel after multiple retry failures.

[0086] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0087] In an embodiment of the present invention, the monitoring system is applied to the monitoring system, which includes at least an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system. In the process of monitoring the indicators, the system can receive indicator events asynchronously published by the target application through the event bus, determine the event type corresponding to the indicator event, and route the indicator event to the target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution process of the target application. Then, the target indicator data corresponding to the indicator event is obtained through the target storage system, and the target indicator data is stored. Then, the target indicator data pushed by the target storage system is received through the monitoring backend, and the target indicator data is displayed. Therefore, in the process of monitoring the application, the event-based model collects indicators, realizes loose coupling between the application and the indicator collection mechanism, and improves the flexibility of indicator collection. At the same time, the scalability of the mechanism can be achieved by introducing the event bus and indicator storage, and batch indicators can be processed at the same time to meet different needs and ensure real-time performance.

[0088] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0089] As an example, see Figure 2 , shows a schematic diagram of the processing flow of indicator data provided in an embodiment of the present invention, which may specifically include:

[0090] 1. The application publishes events containing metric data.

[0091] The present invention collects and publishes the indicator data of the application program in an event-driven manner, avoiding the active pull method in traditional monitoring libraries (such as Micrometer or Prometheus Java Client). Compared with direct indicator collection, the event-based approach can provide greater flexibility and scalability.

[0092] Specific steps:

[0093] Create events containing indicator data: During the execution of an application, various indicator data are generated, such as counters, gauges, or histograms. These indicators are encapsulated into event objects.

[0094] Publishing events: The application publishes these indicator events through the event bus, so that the application and the monitoring system can remain loosely coupled. In this way, the application only needs to focus on business logic without having to deal with the underlying monitoring details.

[0095]

[0096] 2. The event bus routes events to the metrics storage.

[0097] The event bus is one of the core components of the present invention, which is responsible for receiving events published by applications and routing them to appropriate indicator storage systems. As a middleware, the event bus decouples applications from the storage backend, enhancing the flexibility and maintainability of the system.

[0098] Specific steps:

[0099] Event bus listens for events: The event bus listens for events from applications. When an application publishes an event, the event bus automatically captures and processes the event.

[0100] Event routing: The event bus routes the event to the corresponding storage system based on its type and content. For example, if the event is a counter type, the event bus will route it to a storage module dedicated to storing counters. If the event is a measurement value, it will be routed to the measurement value storage module.

[0101]

[0102] 3. Indicator storage stores indicator data in memory.

[0103] In the present invention, the indicator data is first temporarily stored in the memory. This method can efficiently save a large amount of indicator data in a short time and refresh it in batches to the monitoring backend at an appropriate time. The memory storage system is similar to a cache, ensuring that a large amount of indicator data can be quickly processed even in a high-concurrency environment.

[0104] Specific steps:

[0105] Receive and store events: After the indicator storage system receives the event routed by the event bus, it stores the indicator data in the event into a data structure in memory. HashMap or ConcurrentHashMap is usually used to store these indicators so that they can be read and updated quickly.

[0106] Supports real-time updates: The indicator data in the memory storage is updated dynamically. For example, for counter indicators, each time a new event arrives, the count value will be updated without overwriting the original data.

[0107] In this way, it can be ensured that the latest statistical results are stored in the memory.

[0108]

[0109]

[0110] The indicator refresh mechanism in the present invention introduces an indicator refresher to regularly push the indicator data temporarily stored in the memory to the monitoring backend, such as InfluxDB, Prometheus or Grafana. This process is divided into the following specific steps:

[0111] 4.1. Regular refresh interval

[0112] The indicator refresher usually uses a scheduled task or scheduler to control the frequency of indicator refresh. According to actual needs, the common refresh interval is set to every 1 minute, 5 minutes or a custom time interval. For example:

[0113] ScheduledExecutorService scheduler=Executors.newScheduledThreadPool(1);

[0114] scheduler.scheduleAtFixedRate(()->refreshIndicators(),0,1,TimeUnit.MINUTES); / / Refresh every 1 minute

[0115] Default refresh interval: every 1 minute.

[0116] Configurable interval: Developers or operation and maintenance personnel can flexibly configure the refresh interval as needed to meet the needs of different scenarios. For example, in high-concurrency scenarios, the refresh frequency can be increased, while in scenarios with limited resources or low real-time requirements, the refresh interval can be appropriately extended.

[0117] 4.2. Indicator Refresh Mechanism

[0118] During each periodic execution, the indicator refresher extracts all collected indicator data from the indicator storage in memory and pushes them to the monitoring backend in batches. The whole process is as follows:

[0119] Step 1: Read all metrics (may include counters, time series, etc.) from the in-memory store.

[0120] Step 2: Batch these indicators into a request and prepare to send it to the monitoring backend.

[0121] Step 3: Call the API or protocol provided by the monitoring backend (such as Prometheus's HTTP PushGateway and InfluxDB's write interface) to synchronize the indicator data to the monitoring system.

[0122]

[0123] 4.3. How to deal with indicator loss or abnormality

[0124] In order to ensure data integrity, the system will introduce some robust designs to handle possible abnormal situations.

[0125] For example:

[0126] Caching mechanism: When the network connection is abnormal or the monitoring backend is unavailable, the system will temporarily cache the indicator data and try to send it again when the connection is restored.

[0127] Failure retry mechanism: If the refresh operation fails, the system can use an exponential backoff strategy (ExponentialBackoff) to retry to reduce the impact on the system.

[0128]

[0129] 4.4 Customized refresh strategy

[0130] In addition to fixed time intervals, the present invention also allows for custom refresh strategies to adapt to different scenarios. For example: Event-driven refresh: When the indicator data reaches a certain amount or a certain condition is triggered, it can be refreshed immediately to ensure that important indicators can be uploaded in a timely manner. Load-aware refresh: Dynamically adjust the refresh frequency based on the current load or performance of the system. For example, reduce the refresh frequency under high load to reduce system pressure.

[0131] 4.5. Index Refresh and System Performance

[0132] In order not to affect the normal operation of the system, the refresh operation is usually performed asynchronously in a separate thread. This ensures that the collection of indicators and the execution of business logic do not interfere with each other, while improving the scalability of the system.

[0133] In addition, the indicator refresher usually pushes the indicator data in the memory to the monitoring backend regularly every 1 minute or at a custom interval. The refresh process includes reading data from the memory, sending it in batches to the monitoring backend, and supporting caching and retry mechanisms to ensure data integrity. By introducing customized strategies (such as event-driven or load-aware refresh), the system can flexibly adapt to different demand scenarios. Through this mechanism, the present invention can achieve efficient and reliable indicator collection and processing, ensure that the indicator data in the monitoring system is accurate and timely, and at the same time improve the flexibility and scalability of the system.

[0134] Through the above process, based on the loose coupling between the indicator collection mechanism and the application, the application only needs to publish events without directly interacting with the indicator collection mechanism. In addition, by introducing the event bus and indicator storage, the scalability of the mechanism is improved, which can handle a large amount of event and indicator data, and provide customization options, such as custom event format and indicator refresh frequency, to meet specific needs.

[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0136] Reference Figure 3 , shows a structural block diagram of a data processing device provided in an embodiment of the present invention, which is applied to a monitoring system, wherein the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system, and specifically may include the following modules:

[0137] An event receiving module 301 is used to receive an indicator event asynchronously published by a target application through the event bus, determine an event type corresponding to the indicator event, and route the indicator event to a target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution of the target application;

[0138] The storage module 302 is used to obtain the target indicator data corresponding to the indicator event through the target storage system and store the target indicator data;

[0139] The visualization module 303 is used to receive the target indicator data pushed by the target storage system through the monitoring backend and display the target indicator data.

[0140] In some feasible implementations, the event receiving module 301 is specifically used for:

[0141] Obtaining the event type or event content corresponding to the indicator event;

[0142] Determine a target storage system corresponding to the event type or the event content;

[0143] The indicator event is routed to the target storage system.

[0144] In some feasible implementations, the target indicator data includes an indicator name and an indicator value corresponding to the indicator name, and the storage module 302 is specifically used to:

[0145] If the monitoring system runs in a single-threaded environment, determining a first non-thread-safe hash table, and storing the indicator name and the indicator value in the first hash table in the form of a key-value pair;

[0146] If the monitoring system runs in a multi-threaded environment, a thread-safe second hash table is determined, and the indicator name and the indicator value are stored in the second hash table in the form of key-value pairs.

[0147] In some feasible implementations, the storage module 302 is further configured to:

[0148] If the indicator event is a counter type event, then in response to a newly detected counter type event, the target indicator name corresponding to the counter type and the target indicator value corresponding to the target indicator name are determined, and the target indicator value is updated in the first hash table or the second hash table.

[0149] In some feasible implementations, the storage system further includes an indicator refresher located in the storage system, and the device further includes:

[0150] A refresh module, configured to respond to an indicator refresh instruction through the indicator refresher, read the target indicator data from the storage system according to a preset refresh interval, and batch-pack the target indicator data into a data request;

[0151] The push module is used to call the interface or transmission protocol provided by the monitoring backend through the indicator refresher to push the data request to the monitoring backend.

[0152] Some possible implementations also include:

[0153] A cache module is used to respond to the transmission of indicator data to the monitoring backend through the indicator refresher. If a network connection abnormality occurs or the monitoring backend is unavailable during the transmission process, the indicator data being transmitted is cached until the network is restored or the monitoring backend is available, and the indicator data is re-transmitted to the monitoring backend.

[0154] Some possible implementations also include:

[0155] A retry module is used to execute the refresh operation again according to a first preset retry interval if the indicator refresher fails to execute the refresh operation corresponding to the indicator refresh instruction for the first time, wherein after each retry failure, the duration corresponding to the retry interval increases exponentially until the refresh operation is successfully executed.

[0156] In some feasible implementations, the device further includes:

[0157] A response module, configured to read the target indicator data from the storage system and batch-pack the target indicator data into a data request if the indicator refresher detects that the amount of indicator data stored in the storage system reaches a preset threshold or satisfies a preset condition;

[0158] The sending module is used to call the interface or transmission protocol provided by the monitoring backend through the indicator refresher to push the data request to the monitoring backend.

[0159] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0161] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0162] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0163] It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program codes.

[0164] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0168] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0169] The data processing method and the data processing device provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: Applied to a monitoring system, the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system, the method includes: Receiving an index event asynchronously published by a target application through the event bus, determining an event type corresponding to the index event, and routing the index event to a target storage system, wherein the index event is an event obtained by encapsulating target index data generated during the execution of the target application; Acquiring target indicator data corresponding to the indicator event through the target storage system, and storing the target indicator data; The target indicator data pushed by the target storage system is received through the monitoring backend, and the target indicator data is displayed.

2. The method according to claim 1, characterized in that The routing of the indicator event to the target storage system includes: Obtaining the event type or event content corresponding to the indicator event; Determine a target storage system corresponding to the event type or the event content; The indicator event is routed to the target storage system.

3. The method according to claim 1, characterized in that The target indicator data includes an indicator name and an indicator value corresponding to the indicator name, and storing the target indicator data includes: If the monitoring system runs in a single-threaded environment, determining a first non-thread-safe hash table, and storing the indicator name and the indicator value in the first hash table in the form of a key-value pair; If the monitoring system runs in a multi-threaded environment, a thread-safe second hash table is determined, and the indicator name and the indicator value are stored in the second hash table in the form of key-value pairs.

4. The method according to claim 3, characterized in that The storing of the target indicator data further includes: If the indicator event is a counter type event, then in response to a newly detected counter type event, the target indicator name corresponding to the counter type and the target indicator value corresponding to the target indicator name are determined, and the target indicator value is updated in the first hash table or the second hash table.

5. The method according to claim 1, characterized in that The storage system further includes an indicator refresher located in the storage system. After storing the target indicator data, the method further includes: The indicator refresher responds to the indicator refresh instruction and reads the target indicator data from the storage system according to a preset refresh interval, and batch-packs the target indicator data into a data request; The indicator refresher calls the interface or transmission protocol provided by the monitoring backend to push the data request to the monitoring backend.

6. The method according to claim 5, characterized in that Also includes: The indicator refresher responds to transmitting the indicator data to the monitoring backend. If a network connection abnormality occurs or the monitoring backend is unavailable during the transmission process, the indicator data being transmitted is cached until the network is restored or the monitoring backend is available, and the indicator data is then transmitted to the monitoring backend again.

7. The method according to claim 5, characterized in that Also includes: If the indicator refresher fails to execute the refresh operation corresponding to the indicator refresh instruction for the first time, the indicator refresher executes the refresh operation again according to the first preset retry interval; After each failed retry, the retry interval increases exponentially until the refresh operation is successfully performed.

8. The method according to claim 5, characterized in that After storing the target indicator data, the method further includes: If the indicator refresher detects that the amount of indicator data stored in the storage system reaches a preset threshold, or satisfies a preset condition, the target indicator data is read from the storage system, and the target indicator data is batch-packaged into a data request; The indicator refresher calls the interface or transmission protocol provided by the monitoring backend to push the data request to the monitoring backend.

9. A data processing device, characterized in that: Applied to a monitoring system, the monitoring system at least includes an event bus, at least one storage system communicatively connected to the event bus, and a monitoring backend communicatively connected to the storage system, the device includes: An event receiving module is used to receive the indicator event asynchronously published by the target application through the event bus, determine the event type corresponding to the indicator event, and route the indicator event to the target storage system. The indicator event is an event obtained by encapsulating the target indicator data generated during the execution of the target application; A storage module, used to obtain target indicator data corresponding to the indicator event through the target storage system, and store the target indicator data; A visualization module is used to receive the target indicator data pushed by the target storage system through the monitoring backend and display the target indicator data.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 8 when executing the program stored in the memory.

11. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 8.

Images