Data processing methods, apparatus, electronic devices and storage media

Abstract

This application relates to a data processing method, apparatus, electronic device, and storage medium, applied in the field of monitoring technology. The method includes: real-time monitoring of monitoring data collected by a Prometheus instance to obtain update messages of the monitoring data; and sending the update messages of the monitoring data to a data processing program, so that the data processing program stores the monitoring data in a storage server according to the update messages. This application can improve the availability of the Prometheus service.

Description

Technical Field

[0001] This application relates to the field of surveillance technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Prometheus is a well-designed monitoring solution that is widely used. It can monitor objects, collect monitoring data, and store the data locally. As the data volume increases, Prometheus also provides a solution for storing monitoring data remotely. However, if an anomaly occurs in data monitoring or data storage, the entire monitoring service will become unavailable. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a data processing method, apparatus, electronic device and storage medium.

[0004] According to a first aspect of this application, a data processing method is provided, applied to a data monitoring program, comprising:

[0005] The monitoring data collected by the Prometheus instance is monitored in real time, and update messages of the monitoring data are obtained.

[0006] The update message of the monitoring data is sent to the data processing program, so that the data processing program stores the monitoring data in the storage server according to the update message.

[0007] Optionally, the real-time monitoring of the monitoring data collected by the Prometheus instance includes:

[0008] Obtain the configuration file of the Prometheus instance, parse the configuration file to obtain the storage location information of the monitoring data collected by the Prometheus instance; and

[0009] Based on the storage location information, the monitoring data is monitored in real time.

[0010] Optionally, after obtaining the update message of the monitoring data, the method further includes:

[0011] The update message of the monitoring data is sent to the Prometheus main program so that the Prometheus main program can monitor the data listening program.

[0012] According to a second aspect of this application, a data processing method is provided, applied to a data processing program, comprising:

[0013] Receive monitoring data update messages sent by the data monitoring program, wherein the monitoring data update messages are obtained by the data monitoring program through real-time monitoring of the monitoring data collected by the Prometheus instance;

[0014] The monitoring data is stored in the storage server according to the update message of the monitoring data.

[0015] Optionally, after receiving the update message of the monitoring data, the method further includes:

[0016] Based on the update messages of the monitored data, the data processing pressure is determined; and

[0017] Based on the data processing pressure, the system dynamically adjusts its resource allocation.

[0018] Optionally, after dynamically adjusting its resource configuration according to the data processing pressure, the method further includes:

[0019] The data processing pressure and resource adjustment results are sent to the Prometheus main program so that the Prometheus main program can monitor the data processing program.

[0020] Optionally, if the number of Prometheus instances is multiple;

[0021] The step of storing the monitoring data to the storage server according to the update message of the monitoring data includes:

[0022] Based on the update message of the monitoring data, the monitoring data collected by multiple Prometheus instances is stored in the storage server using a distributed lock.

[0023] Optionally, receiving update messages for the monitoring data includes:

[0024] The monitoring data update messages are received via a message queue.

[0025] According to a third aspect of this application, a data processing apparatus is provided, comprising:

[0026] The data monitoring module is used to monitor the monitoring data collected by the Prometheus instance in real time and obtain update messages of the monitoring data.

[0027] The first message sending module is used to send the update message of the monitoring data to the data processing program, so that the data processing program stores the monitoring data to the storage server according to the update message of the monitoring data.

[0028] Optionally, the data monitoring module is specifically used to obtain the configuration file of the Prometheus instance, parse the configuration file to obtain the storage location information of the monitoring data collected by the Prometheus instance; and, based on the storage location information, to monitor the monitoring data in real time and obtain update messages of the monitoring data.

[0029] Optionally, the device further includes:

[0030] The second message sending module is used to send the update message of the monitoring data to the Prometheus main program, so that the Prometheus main program can monitor the data listening program.

[0031] According to a fourth aspect of this application, a data processing apparatus is provided, comprising:

[0032] The message receiving module is used to receive update messages of monitoring data sent by the data listening program. The update messages of monitoring data are obtained by the data listening program through real-time monitoring of the monitoring data collected by the Prometheus instance.

[0033] The data processing module is used to store the monitoring data to the storage server according to the update message of the monitoring data.

[0034] Optionally, the device further includes:

[0035] A data processing pressure determination module is used to determine the data processing pressure based on the update messages of the monitored data; and

[0036] The dynamic adjustment module is used to dynamically adjust its own resource configuration according to the data processing pressure.

[0037] Optionally, the device further includes:

[0038] The data sending module is used to send the data processing pressure and resource adjustment results to the Prometheus main program, so that the Prometheus main program can monitor the data processing program.

[0039] Optionally, if there are multiple Prometheus instances, the data processing module is specifically used to store the monitoring data collected by the multiple Prometheus instances to the storage server using a distributed lock, based on the update message of the monitoring data.

[0040] Optionally, the message receiving module is specifically used to receive update messages of the monitoring data in the form of a message queue.

[0041] According to a fifth aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first or second aspect.

[0042] According to a sixth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or second aspect.

[0043] According to a seventh aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first or second aspect.

[0044] The technical solution provided in this application has the following advantages compared with the prior art:

[0045] The data listening program monitors the monitoring data collected by the Prometheus instance in real time, obtains update messages, and sends these messages to the data processing program. The data processing program then stores the monitoring data on the storage server based on these updates. Since the data listening program and the data processing program are independent programs, a crash in either program will not affect the normal service of the other, nor will it affect the service of the main Prometheus program. This provides better fault tolerance and a more stable service guarantee, thus improving the availability of the Prometheus service. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a system architecture applicable to the data processing method in the embodiments of this application;

[0049] Figure 2A A flowchart illustrating the data processing method implemented in this application is provided.

[0050] Figure 2BAnother flowchart illustrating the data processing method implemented in this application;

[0051] Figure 3 An interactive flowchart illustrating the data processing method implemented in this application is provided.

[0052] Figure 4A This application provides a schematic diagram of the structure of a data processing device.

[0053] Figure 4B This is a schematic diagram of a data processing device implemented in this application;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0055] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0056] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0057] Figure 1 This is a schematic diagram of a system architecture applicable to the data processing method in the embodiments of this application, such as... Figure 1 As shown, the system architecture 100 includes a cluster 110, a network 120, and a storage server 130 (e.g., an object storage server). The network 120 serves as the medium for providing a communication link between the cluster 110 and the storage server 130. The network 120 can include various connection types, such as wired or wireless communication links or fiber optic cables. A Prometheus instance can be deployed in the cluster 110. Through the Prometheus instance, monitoring objects 101 and 102 can be performed to obtain monitoring data. This application can use a data listening program to monitor the monitoring data collected by the Prometheus instance in real time, obtain update messages of the monitoring data, and send the update messages to a data processing program. The data processing program can then store the monitoring data in the storage server 130 according to the update messages.

[0058] This application implements a method for backing up and storing Prometheus monitoring data to a storage server through a data monitoring program and a data processing program. Since it is not coupled with the main Prometheus program, data monitoring and data storage do not affect each other's normal service, thereby improving the availability of Prometheus.

[0059] See Figure 2A , Figure 2A A flowchart illustrating a data processing method implemented in this application, applied to a data monitoring program, may include the following steps:

[0060] Step S202: Monitor the monitoring data collected by the Prometheus instance in real time and obtain update messages for the monitoring data.

[0061] Prometheus is a well-designed and widely adopted system monitoring solution. By collecting data exposed by monitored nodes and providing a query interface, it allows maintenance personnel to observe multi-dimensional system status. In this embodiment, Prometheus instances can be deployed in a cluster to monitor the monitored objects. Clusters can be deployed in different regions, allowing monitoring of monitored objects in different regions to be achieved through clusters in different regions.

[0062] Each monitored object can expose the collected monitoring data through a local HTTP (Hypertext Transfer Protocol) service. The Prometheus instance is responsible for collecting and storing this monitoring data. Because Prometheus has a rich data model, it can effectively describe data from various scenarios. The monitored object can be any machine or device to be monitored, and the monitoring data refers to the data obtained from monitoring the monitored object, which can include monitoring metrics and their magnitude. For example, if the monitored object is a server, the monitoring metrics could be the server's CPU utilization, memory usage, etc.

[0063] It is understandable that the size of monitoring metric data varies at different times; therefore, the monitoring data collected by Prometheus instances is time-series data. Prometheus instances can store the collected monitoring data locally. However, since local storage space is typically limited, Prometheus instances can store short-term monitoring data but cannot store longer-term monitoring data. To enable long-term storage of monitoring data, this application can store the monitoring data on a remote storage server (e.g., in the cloud). For example, it can store it in a time-series database on the storage server. Time-series databases are specifically designed for storing and managing time-series data, offering better support for time-series data compared to traditional relational databases, while providing high-performance read / write and computation capabilities.

[0064] In this embodiment, a data monitoring program, independent of the main Prometheus program, can be provided to monitor the monitoring data collected by the Prometheus instance in real time, ensuring the timeliness of monitoring data collection and storage. Specifically, during the process of the Prometheus instance storing monitoring data locally, real-time monitoring can be performed to obtain update messages of the monitoring data. Update messages of the monitoring data may include: the update time of the monitoring data, the updated monitoring data, the size of the updated monitoring data, etc.

[0065] Step S204: Send the update message of the monitoring data to the data processing program so that the data processing program stores the monitoring data to the storage server according to the update message.

[0066] After the data listening program receives update messages from the monitoring data, it can send these messages to the data processing program. Similar to the data listening program, the data processing program is also independent of the main Prometheus program and is used to store the monitoring data on the storage server. The data processing program stores the monitoring data on the storage server based on the received update messages. It can retry in case of failure and issue timely alerts.

[0067] As can be seen, the data monitoring program and the data processing program are independent, implementing data monitoring and data storage functions respectively. By combining the two, monitoring data can be stored on a remote storage server. Furthermore, both can be maintained and upgraded independently without affecting the other's normal service, and support for different time-series databases on different storage servers can be easily extended.

[0068] It should be noted that the data monitoring program and the data processing program can be deployed on the same device or on different devices. If deployed on different devices, cross-device shared access can be achieved through the file system.

[0069] The data processing method in this embodiment of the application uses a data monitoring program to monitor the monitoring data collected by the Prometheus instance in real time, obtain update messages of the monitoring data, and send the update messages to the data processing program. The data processing program then stores the monitoring data in the storage server based on the update messages. Since the data monitoring program and the data processing program are independent programs, a crash in either program will not affect the normal service of the other, nor will it affect the service of the main Prometheus program. This provides better fault tolerance and a more stable service guarantee, thus improving the availability of the Prometheus service.

[0070] See Figure 2B , Figure 2B This application provides another flowchart of the data processing method, which, when applied to a data processing program, may include the following steps:

[0071] Step S210: Receive the monitoring data update message sent by the data listening program. The monitoring data update message is obtained by the data listening program through real-time monitoring of the monitoring data collected by the Prometheus instance.

[0072] Step S220: Store the monitoring data to the storage server according to the update message of the monitoring data.

[0073] Figure 2B The embodiment is with Figure 2A The data processing method on the data processing program side corresponding to the embodiment is described in Figure 2, and will not be repeated here.

[0074] See Figure 3 , Figure 3 An interactive flowchart illustrating the data processing method implemented in this application may include the following steps:

[0075] Step S302: The data monitoring program listens to the monitoring data collected by the Prometheus instance in real time and obtains the update messages of the monitoring data.

[0076] In this embodiment of the application, the method for the data monitoring program to monitor the monitoring data in real time may be as follows: the data monitoring program obtains the configuration file of the Prometheus instance, which contains the configuration information after the maintenance personnel configure the Prometheus instance. The configuration information may include: the indicator data to be monitored, the start time of monitoring, the end time of monitoring, and the storage location of the obtained monitoring data, etc.

[0077] The data monitoring program can obtain the storage location information of the monitoring data collected by the Prometheus instance by parsing the configuration file; and monitor the monitoring data in real time based on the storage location information.

[0078] In step S304, the data monitoring program sends the update message of the monitored data to the Prometheus main program.

[0079] Because the data listener is a separate program, independent of the main Prometheus program, the update message can be sent to the main Prometheus program. This allows the main Prometheus program to monitor the data listener, thereby improving the availability of the Prometheus service.

[0080] In step S306, the Prometheus main program monitors the data listening program.

[0081] For example, if the data monitoring program malfunctions, the Prometheus main program can be notified in time and switch to the Prometheus main program to execute the monitoring process. After the data monitoring program recovers, it can switch back to the data monitoring program to execute the monitoring process.

[0082] In step S308, the data monitoring program sends the update message of the monitored data to the data processing program.

[0083] In step S310, the data processing program determines the data processing pressure based on the update messages of the monitoring data, and dynamically adjusts its own resource configuration according to the data processing pressure.

[0084] In this embodiment, the data processing program can receive monitoring data update messages via message queues, specifically one or more message queues. Based on the received monitoring data update messages, the data processing pressure can be determined. The higher the reception speed of the monitoring data update messages, the greater the data processing pressure; similarly, the larger the size of the monitoring data, the greater the data processing pressure. For example, during peak business hours (e.g., morning and afternoon), more monitoring data is collected, resulting in higher data processing pressure; conversely, during off-peak business hours (e.g., early morning), less data is collected, leading to lower data processing pressure.

[0085] Based on data processing pressure, data processing programs can dynamically adjust their resource configuration to optimize resource usage. The resource configuration of a data processing program can include its own memory usage, the size of its central processing unit (CPU), and the number of data queues it processes. Specifically, under high data processing pressure, the number of message queues can be increased, and correspondingly, the number of data queues processed by the data processing program will also increase. And / or, the memory usage can be increased, and / or, the CPU size can be increased, etc. Conversely, under low data processing pressure, the number of message queues can be decreased, and correspondingly, the number of data queues processed by the data processing program will decrease. And / or, the memory usage can be reduced, and / or, the CPU size can be reduced, etc.

[0086] In step S312, the data processing program stores the monitoring data to the storage server based on the update message of the monitoring data.

[0087] The update message for monitoring data includes information such as the storage location of the monitoring data, the update time, the updated monitoring data, and the size of the updated monitoring data. The data processing program can store the monitoring data to the storage server based on the update message. In an optional implementation, the data processing program can support etcd and Kubernetes, which enable high availability of data storage.

[0088] etcd is a distributed, highly available, and consistent key-value store database for shared configuration and service discovery. Implemented in Go, it stores critical data in distributed systems. Kubernetes is an open-source application management system for managing containerized applications across multiple hosts in a cloud platform. Kubernetes aims to make deploying containerized applications simple and efficient, providing a mechanism for application deployment, planning, updating, and maintenance.

[0089] It's worth noting that if there are multiple Prometheus instances, all instances can monitor the object being monitored and obtain monitoring data. The data processing program can then use distributed locks to store the monitoring data collected by multiple Prometheus instances on a storage server based on the update messages.

[0090] Distributed locks are a method for controlling synchronized access to shared resources between distributed systems. In distributed systems, coordination of their actions is often necessary. If different systems or different hosts within the same system share one or more resources, mutual exclusion can be used to prevent interference and ensure consistency when accessing these resources. In this embodiment, the data processing program can utilize distributed locks to synchronize the running state of each Prometheus instance. When one Prometheus instance encounters an error, other Prometheus instances can automatically take over, continuing to execute the data processing program and improving the availability of the Prometheus service.

[0091] In step S314, the data processing program sends the data processing pressure and resource adjustment results to the Prometheus main program.

[0092] Step S316: The Prometheus main program monitors the data processing program.

[0093] Similar to the aforementioned data monitoring program, the data processing program can also send its data processing pressure and resource adjustment results to the Prometheus main program. The Prometheus main program can then know the current data processing status of the data processing program. For example, if the data processing pressure is too high and causes an anomaly, it can switch to the Prometheus main program to continue execution. After the data processing program returns to normal, it can switch back to the data processing program to continue execution, ensuring the availability of the Prometheus service.

[0094] In addition, after receiving an update message for the monitoring data, the data processing program can forward the update message to other programs, which can then perform corresponding operations based on the message.

[0095] The data processing method in this embodiment of the application uses a data listening program to monitor the monitoring data collected by the Prometheus instance in real time, obtain update messages of the monitoring data, and send the update messages to a data processing program. The data processing program can then store the monitoring data in a storage server based on the update messages. Since the data listening program and the data processing program are independent programs, the crash of either program does not affect the normal service of the other, nor does it affect the service of the main Prometheus program. This provides better fault tolerance and more stable service guarantees, thus improving the availability of Prometheus. The data processing program can also dynamically adjust its resource configuration according to the current data processing pressure, thereby optimizing resource utilization. Both the data listening program and the data processing program can send their status to the main Prometheus program, which monitors both. Furthermore, when multiple Prometheus instances are deployed, a distributed lock is used to synchronize the status of multiple Prometheus instances to further ensure the availability of the Prometheus service.

[0096] Corresponding to the above method embodiments, this application also provides a data processing apparatus, see below. Figure 4A , Figure 4A This application provides a schematic diagram of a data processing apparatus, comprising:

[0097] The data monitoring module 402 is used to monitor the monitoring data collected by the Prometheus instance in real time and obtain the update messages of the monitoring data.

[0098] The first message sending module 404 is used to send update messages of monitoring data to the data processing program, so that the data processing program can store the monitoring data to the storage server according to the update messages of the monitoring data.

[0099] Optionally, the data monitoring module 402 is specifically used to obtain the configuration file of the Prometheus instance, parse the configuration file to obtain the storage location information of the monitoring data collected by the Prometheus instance; and, based on the storage location information, to monitor the monitoring data in real time and obtain update messages of the monitoring data.

[0100] Optionally, the above data processing apparatus further includes:

[0101] The second message sending module is used to send update messages of monitoring data to the Prometheus main program, so that the Prometheus main program can monitor the data listening program.

[0102] See Figure 4B , Figure 4B This application provides another schematic diagram of the structure of a data processing device, including:

[0103] The message receiving module 410 is used to receive update messages of monitoring data sent by the data listening program. The update messages of monitoring data are obtained by the data listening program through real-time monitoring of the monitoring data collected by the Prometheus instance.

[0104] The data processing module 420 is used to store the monitoring data to the storage server based on the update messages of the monitoring data.

[0105] Optionally, the above data processing apparatus further includes:

[0106] The data processing pressure determination module is used to determine the data processing pressure based on the update messages of the monitoring data; and

[0107] The dynamic adjustment module is used to dynamically adjust its own resource configuration based on data processing pressure.

[0108] Optionally, the above data processing apparatus further includes:

[0109] The data sending module is used to send data processing pressure and resource adjustment results to the Prometheus main program, so that the Prometheus main program can monitor the data processing program.

[0110] Optionally, if there are multiple Prometheus instances, the data processing module 420 is specifically used to store the monitoring data collected by multiple Prometheus instances to the storage server using a distributed lock based on the update messages of the monitoring data.

[0111] Optionally, the message receiving module 410 is specifically used to receive update messages of monitoring data in the form of a message queue.

[0112] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0113] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0114] In an exemplary embodiment of this application, an electronic device is also provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the data processing method described in this exemplary embodiment.

[0115] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. It should be noted that... Figure 5 The electronic device 500 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0116] like Figure 5 As shown, the electronic device 500 includes a central processing unit 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The central processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network (LAN) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0118] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit 501, it performs various functions defined in the system of this application.

[0119] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-described data processing method.

[0120] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0121] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the above-described data processing method.

[0122] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, The method, applied to a data monitoring program, includes: During the process of the Prometheus instance storing monitoring data locally, the monitoring data collected by the Prometheus instance is monitored in real time to obtain update messages of the monitoring data. The update messages include the update time of the monitoring data, the updated monitoring data, and the size of the updated monitoring data. The update message of the monitoring data is sent to the data processing program, so that the data processing program stores the monitoring data to the storage server according to the update message of the monitoring data. The data monitoring program and the data processing program are independent programs. After obtaining the update message of the monitoring data, the method further includes: The update message of the monitoring data is sent to the Prometheus main program so that the Prometheus main program can monitor the data listening program. When the data listening program malfunctions, the monitoring process is switched to the Prometheus main program. After the data listening program recovers, the monitoring process is switched to the data listening program.

2. The method according to claim 1, characterized in that, The real-time monitoring of the data collected from the Prometheus instance includes: Obtain the configuration file of the Prometheus instance, parse the configuration file to obtain the storage location information of the monitoring data collected by the Prometheus instance; and Based on the storage location information, the monitoring data is monitored in real time.

3. A data processing method, characterized in that, Applied to a data processing program, the method includes: The system receives monitoring data update messages sent by a data monitoring program. These update messages are obtained by the data monitoring program through real-time monitoring of the monitoring data collected by the Prometheus instance during the process of the Prometheus instance storing the monitoring data locally. The update messages include the update time of the monitoring data, the updated monitoring data, and the size of the updated monitoring data. The data monitoring program and the data processing program are independent programs. The monitoring data is stored in the storage server according to the update message of the monitoring data. After receiving the update message of the monitoring data, the method further includes: Based on the update messages of the monitored data, the data processing pressure is determined; and Based on the data processing pressure, dynamically adjust its own resource allocation; The data processing pressure and resource adjustment results are sent to the Prometheus main program so that the Prometheus main program can monitor the data processing program. If the data processing program malfunctions, the main program will switch to the Prometheus main program to process the data. After the data processing program returns to normal, the main program will switch back to the data processing program to process the data.

4. The method according to claim 3, characterized in that, If the number of Prometheus instances is multiple; The step of storing the monitoring data to the storage server according to the update message of the monitoring data includes: Based on the update message of the monitoring data, the monitoring data collected by multiple Prometheus instances is stored in the storage server using a distributed lock.

5. The method according to claim 3, characterized in that, Receiving update messages for the monitoring data includes: The monitoring data update messages are received via a message queue.

6. A data processing apparatus, characterized in that, The device includes: The data monitoring module is used to monitor the monitoring data collected by the Prometheus instance in real time during the process of the Prometheus instance storing the monitoring data locally, and to obtain the update message of the monitoring data. The update message includes the update time of the monitoring data, the updated monitoring data, and the size of the updated monitoring data. The first message sending module is used to send the update message of the monitoring data to the data processing program, so that the data processing program stores the monitoring data to the storage server according to the update message of the monitoring data. The data listening program and the data processing program are independent programs. The second message sending module is used to send the update message of the monitoring data to the Prometheus main program, so that the Prometheus main program can monitor the data listening program. When the data listening program encounters an abnormality, the module switches to the Prometheus main program to perform the listening process. After the data listening program recovers, the module switches back to the data listening program to perform the listening process.

7. A data processing apparatus, characterized in that, The device includes: The message receiving module is used to receive update messages of monitoring data sent by the data listening program. The update messages of monitoring data are obtained by the data listening program in real time by listening to the monitoring data collected by the Prometheus instance during the process of the Prometheus instance storing the monitoring data locally. The update messages include the update time of the monitoring data, the updated monitoring data, and the size of the updated monitoring data. The data listening program and the data processing program are independent programs. The data processing module is used to store the monitoring data to the storage server according to the update message of the monitoring data; The data processing pressure determination module is used to determine the data processing pressure based on the update messages of the monitoring data. The dynamic adjustment module is used to dynamically adjust its own resource configuration according to the data processing pressure. The data sending module is used to send the data processing pressure and resource adjustment results to the Prometheus main program, so that the Prometheus main program can monitor the data processing program, switch to the Prometheus main program to process data when the data processing program is abnormal, and switch back to the data processing program to process data after the data processing program returns to normal.

8. An electronic device, characterized in that, include: A processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-2, or implements the steps of the method according to any one of claims 3-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-2, or the steps of the method according to any one of claims 3-5.

Images