Methods and devices for processing work monitoring data, storage media, and electronic equipment
By pushing Flink job monitoring data to a message queue partition and pulling it from the data analysis library when needed, the problem of high pressure on the monitoring data acquisition end is solved, achieving stable acquisition and efficient data processing.
Patent Information
- Application Number
- CN202211148968.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-21
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-09-21
AI Technical Summary
With an increasing number of Flink jobs, the monitoring data acquisition end directly pulls monitoring data from the processing engine, resulting in high pressure, high failure rate, and inability to process data in a timely manner, which can easily lead to the loss of monitoring data.
By pushing job monitoring data to a message queue partition and determining the address of the data analysis database based on the job identifier when needed, monitoring data is pulled from the data analysis database. Stable acquisition is achieved by using the queue relay interface and the data analysis database to share the address of the network node device.
It improved the efficiency of acquiring operation monitoring data, reduced the pressure of acquiring monitoring data, reduced data loss and errors, and lowered the failure rate.
Smart Images

Figure CN115562933B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data processing, and more specifically, to a method and apparatus for processing operational monitoring data, a storage medium, and an electronic device. Background Technology
[0002] In big data processing, Flink, as a streaming and batch data processing framework, is widely used, and many businesses have begun to adopt Flink as their data processing engine. However, with the increase in Flink data processing engine jobs or tasks, monitoring Flink data processing engine jobs has also become a challenge.
[0003] Flink jobs primarily involve streaming data processing and run for extended periods. As a stream processing framework, it includes a relatively comprehensive monitoring mechanism and provides an interface through the Flink WebMonitor monitoring data acquisition end for external systems to obtain monitoring data. Therefore, many existing monitoring systems directly obtain monitoring data from Flink jobs for further processing and analysis, or for direct storage and display, etc. Figure 1 As shown, Figure 1 This is a flowchart of existing methods for retrieving job monitoring data. However, as the number of running jobs in Flink increases, the amount of job monitoring data retrieved by the monitoring data acquisition end becomes very large, putting significant pressure on the acquisition end. If not processed promptly, monitoring data loss can easily occur. Furthermore, once a job finishes running or encounters an anomaly, it becomes impossible to access the interface to retrieve monitoring data, including historical monitoring data.
[0004] In the context of Flink, a big data processing engine, the increasing number of monitoring data leads to a surge in job volume. The method by which the monitoring data acquisition end directly pulls monitoring data from the processing engine puts significant pressure on the monitoring data acquisition end, resulting in a high failure rate. Currently, no effective solution has been proposed. Summary of the Invention
[0005] The main objective of this application is to provide a method, apparatus, storage medium, and electronic device for processing job monitoring data, in order to solve the problem in related technologies where, as the number of jobs increases, the monitoring data increases, and the monitoring data acquisition end directly pulls the monitoring data from the processing engine, which puts a lot of pressure on the monitoring data acquisition end and causes a high failure rate.
[0006] To achieve the above objectives, according to one aspect of this application, a method for processing job monitoring data is provided, comprising: determining a queue partition identifier corresponding to the job monitoring data of a target job based on the job identifier of the target job, wherein the target job is a job for which monitoring data needs to be obtained, and the job monitoring data is stored in a uniquely corresponding data analysis library through the corresponding queue partition; determining the address of the corresponding data analysis library based on the queue partition identifier; and sending a data query request to the data analysis library based on the address to retrieve the job monitoring data of the target job from the data analysis library.
[0007] As an optional approach, determining the address of the corresponding data analysis library based on the queue partition identifier includes: determining the corresponding queue relay interface identifier based on the queue partition identifier, wherein the queue relay interface corresponds one-to-one with the queue partition, and the queue relay interface is used to pull the corresponding job monitoring data from the queue partition and push it to the corresponding data analysis library; and using the address of the queue relay interface corresponding to the queue relay interface identifier as the address of the corresponding data analysis library, wherein the queue relay interface and the corresponding data analysis library are set on the same network node device and share the address of the network node device.
[0008] As an optional approach, before sending a data query request to the data analysis library based on the address and pulling the job monitoring data of the target job from the data analysis library, the method further includes: pulling the job monitoring data of the corresponding job from the corresponding queue transfer interface through the data analysis library at a preset speed; and storing the pulled job monitoring data in the data analysis library in sequence.
[0009] As an optional approach, before retrieving the job monitoring data of the corresponding job from the corresponding queue transfer interface at a preset speed through the data analysis library, the method further includes: consuming the job monitoring data of the corresponding job stored in the corresponding queue partition through the queue transfer interface, wherein multiple queue transfer interfaces corresponding to multiple different queue partitions consume data from the corresponding queue partitions through the same consumption group; and caching the consumed job monitoring data in the cache of the queue transfer interface.
[0010] As an optional approach, before the data analysis library retrieves the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed, the method further includes: if the data analysis library does not retrieve data from the queue transfer interface within a first preset time, pausing the queue transfer interface from consuming job monitoring data from the corresponding queue partition; and restarting the queue transfer interface from consuming job monitoring data from the corresponding queue partition when the data analysis library resumes retrieving data from the queue transfer interface.
[0011] As an optional solution, the method further includes: pushing alarm data from the job monitoring data to an alarm manager through the data analysis library, wherein the alarm data is a portion of the job monitoring data that needs to be detected for alarms, and the alarm manager, the queue transfer interface, and the corresponding data analysis library are located on the same network node device and share the address of the network node device; receiving alarm information pushed by the alarm manager, wherein the alarm manager is used to analyze the alarm data in the data analysis library according to preset rules to generate the alarm information.
[0012] As an optional solution, the method further includes: if no heartbeat message is received from the queue relay interface within a second preset time, determining that the network node device is down, creating a new network node device based on the address of the network node device; creating a new queue relay interface in the new network node device based on the queue relay interface identifier, as well as a new data analysis library and a new alarm manager corresponding to the new queue relay interface, as backup components, and consuming the job monitoring data from the queue partition.
[0013] As an optional approach, before determining the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job, the method further includes: determining the available queue partitions in the message queue, wherein the message queue includes multiple queue partitions, and the available queue partition is at least one partition selected from the multiple queue partitions according to a preset storage principle; in the case of creating the target job, generating the job identifier of the target job according to the available queue partition identifier of the available queue partition corresponding to the target job through a preset algorithm.
[0014] As an optional approach, determining the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job includes: determining the corresponding available queue partition identifier by performing an inverse operation using the preset algorithm based on the job identifier of the target job; and determining the corresponding queue partition identifier based on the available queue partition identifier, wherein the available queue partition identifier corresponds to the queue partition identifier in the message queue.
[0015] To achieve the above objectives, according to another aspect of this application, a processing apparatus for job monitoring data is provided, comprising: a determining module, configured to determine a queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job, wherein the target job is a job for which monitoring data needs to be acquired, and the job monitoring data is stored in a uniquely corresponding data analysis library through the corresponding queue partition; an address module, configured to determine the address of the corresponding data analysis library based on the queue partition identifier; and a fetching module, configured to send a data query request to the data analysis library based on the address, and fetch the job monitoring data of the target job from the data analysis library.
[0016] According to another aspect of this application, a computer-readable storage medium is also provided, the storage medium being used to store a program, wherein the program executes the job monitoring data processing method described in any one of the foregoing claims.
[0017] According to another aspect of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the job monitoring data processing method described in any one of the above.
[0018] This application pushes job monitoring data from the job system to a message queue partition, and then pulls it into a data analysis database. This achieves stable acquisition of job monitoring data. When a target job is needed, the address of the corresponding data analysis database is determined based on the job identifier of the target job, and the corresponding job monitoring data is retrieved from the data analysis database. This achieves the goal of pulling job monitoring data on demand, improving the efficiency of job monitoring data acquisition, reducing the pressure on monitoring data acquisition, eliminating the need to continuously pull and store all job monitoring data, and correspondingly reducing the loss and errors of job monitoring data, thus lowering the failure rate of job monitoring data retrieval. Furthermore, it solves the problem in related technologies where, with the increase in the number of jobs, the monitoring data acquisition end directly pulls monitoring data from the processing engine, which puts a lot of pressure on the monitoring data acquisition end and causes a high failure rate. Attached Figure Description
[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0020] Figure 1 This is a flowchart of existing methods for retrieving job monitoring data;
[0021] Figure 2 This is a flowchart of a method for processing job monitoring data according to an embodiment of this application;
[0022] Figure 3 This is a flowchart of a job monitoring data retrieval method provided in an embodiment of this application;
[0023] Figure 4 This is a flowchart of a backup method for retrieving job monitoring data according to an embodiment of this application;
[0024] Figure 5 This is a flowchart illustrating the process of retrieving job monitoring data from the monitoring data acquisition terminal provided in the embodiments of this application;
[0025] Figure 6 This is a schematic diagram of a work monitoring data processing device according to an embodiment of this application;
[0026] Figure 7 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0030] The present invention will now be described in conjunction with preferred implementation steps. Figure 2 This is a flowchart of a method for processing job monitoring data according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:
[0031] Step S201: Determine the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job. The target job is the job for which monitoring data needs to be obtained. The job monitoring data is stored in a unique data analysis database through the corresponding queue partition.
[0032] Step S202: Determine the address of the corresponding data analysis database based on the queue partition identifier;
[0033] Step S203: Based on the address, send a data query request to the data analysis database and retrieve the job monitoring data of the target job from the data analysis database.
[0034] The entity executing the above steps can be the monitoring data acquisition terminal of the job system. The job system can be a large data processing engine, such as Flink, and the monitoring data acquisition terminal can be used to acquire job monitoring data from different jobs within the job system. Each job in the job system can be a processing stage or sub-process of task data, during which some data needs to be monitored, such as processing speed, resource consumption, and status. In existing technologies, the monitoring data acquisition terminal is directly connected to the job system, directly pulling job monitoring data from all running jobs. Once the workload of the job system increases, the data processing and storage pressure on the monitoring data acquisition terminal will increase significantly. Under hardware or limited resource constraints, the monitoring data acquisition terminal may easily experience data loss or failure.
[0035] Through the above steps, the job monitoring data of the job system is pushed to the message queue partition and then pulled into the data analysis database. This achieves stable acquisition of job monitoring data. When a target job needs to be retrieved, the address of the corresponding data analysis database is determined based on the job identifier of the target job, and the corresponding job monitoring data is retrieved from the data analysis database. This achieves the goal of retrieving job monitoring data on demand, improving the efficiency of job monitoring data acquisition, reducing the pressure on monitoring data acquisition, eliminating the need to continuously retrieve and store all job monitoring data, and correspondingly reducing the loss and error of job monitoring data, thus lowering the failure rate of job monitoring data retrieval. This solves the problem in related technologies where, with the increase in the number of jobs, the monitoring data acquisition end directly pulls monitoring data from the processing engine, which puts a lot of pressure on the monitoring data acquisition end and causes a high failure rate.
[0036] From the perspective of the monitoring data acquisition end, the system first receives the job identifier of the target job for which monitoring data needs to be acquired, and then determines the corresponding queue partition identifier based on the job identifier. The queue partition identifier corresponds directly or indirectly to the data analysis database. The job identifier can be the job ID, and the queue partition identifier can be the queue partition number.
[0037] Generally, queue partitions correspond indirectly to data analysis libraries through corresponding queue transfer interfaces. That is, the queue partition identifier corresponds to the queue transfer interface identifier, and the queue transfer interface identifier corresponds to the data analysis library identifier. Therefore, based on the target job's job identifier, the queue transfer interface and data analysis library for processing the target job's monitoring data can be determined. The target job's monitoring data is then stored in the data analysis library, facilitating subsequent retrieval of the target job's monitoring data.
[0038] It's important to note that queue partitions are created by message queues. To improve task processing efficiency, job systems divide tasks into multiple jobs for parallel processing. Specifically, message queues can be used to assign different jobs to different queue partitions for processing.
[0039] Figure 3 This is a flowchart of the job monitoring data retrieval method provided in the embodiments of this application, such as... Figure 3 As shown, when a queue partition is created, a queue relay interface (Kafka Reporter) and a data analysis library (Prometheus) are created simultaneously for that queue partition. The queue relay interface pulls and caches the job monitoring data of the queue partition, and then the data analysis library pulls the job monitoring data from the queue relay interface and stores it in the data analysis library so that it can respond to the requests from the monitoring data acquisition end and allow the monitoring data end to pull the job monitoring data of the requested target job from the data analysis library.
[0040] Kafka is an open-source distributed message queue. A Kafka Topic is a logical concept within Kafka used to organize and manage a type of message (data). Each piece of data in Kafka belongs to a specific Topic. A Partition is a sub-concept within a Topic, essentially dividing the Topic into chunks to improve throughput (increasing parallel read and write capabilities). Each Topic has at least one replica for actual data storage and disaster recovery. Prometheus, mentioned above, is an open-source monitoring data collection, analysis, and alerting system.
[0041] It should be noted that the Kafka Reporter described above is a program developed according to the Promtheus interface specification that can consume Kafka data and provide an interface for Promtheus to retrieve data. This differs from existing Reporter technologies that can only monitor Kafka's operational metrics.
[0042] Because the queue relay interface can cache job monitoring data and dynamically adjust the speed at which it retrieves job monitoring data from the queue partition based on the data analysis library's retrieval status, it enables dynamic adjustment of resources for acquiring job monitoring data. Meanwhile, the data analysis library retrieves job monitoring data from the queue relay interface at a certain speed, maintaining the stability of the data retrieval process and ensuring stable acquisition of job monitoring data.
[0043] Optionally, before step S201 determines the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job, the method further includes: determining the available queue partitions in the message queue, wherein the message queue includes multiple queue partitions, and the available queue partition is at least one partition selected from the multiple queue partitions according to a preset storage principle; in the case of creating a target job, generating the job identifier of the target job based on the available queue partition identifier of the available queue partition corresponding to the target job through a preset algorithm.
[0044] The job identifier is generated during job creation. During job creation, the available queue partitions in the message queue are first determined, and their numbers are counted. It is then determined which available queue partition the created job should be processed by. Based on the available queue partition identifier (here, the available queue partition number, such as 0, 1, 2, etc.), a preset algorithm is used to calculate and generate the job identifier. This establishes a mapping between the job identifier and the available queue partition. In practice, the job identifier can be a job ID, which can be a fixed-length random large integer, such as 1234567890123, etc. An example is shown in Table 1, which represents the correspondence between job IDs and queue partition identifiers.
[0045] Table 1. Correspondence between Job ID and Queue Partition Identifier
[0046] Assignment ID Queue partition identifier 1234567890123 0 1234567890124 1 1234567890125 2 1234567890126 0 1234567890127 1 1234567890128 2 … …
[0047] The available queue partitions here refer to the queue partitions among the multiple queue partitions of the message queue that are needed to store job monitoring data. The relationship between the number of available queue partitions K and the actual number of queue partitions N in the message queue is K <= N, where <= means less than or equal to. The number of available queue partitions K can be increased or decreased according to demand, while the number of actual queue partitions N is usually increased only according to demand. The purpose is to centralize job monitoring data to a few specific queue partitions, thereby saving computing or network resources on other nodes and making full use of the resources of the nodes currently in use. The above-mentioned preset selection principle can be at least one of the queue partitions currently in use, and factors such as the amount of job data and the remaining resources of the queue partition can be considered when making the selection.
[0048] For example, the message queue mentioned above has three queue partitions, 0, 1, and 2, all of which are used as available queue partitions, resulting in a total of 3 available queue partitions. Job monitoring data will be distributed sequentially to these three available queue partitions. As shown in Table 1, multiple jobs are written to available queue partitions sequentially. Currently running jobs (only running jobs generate monitoring data) focus on sending monitoring data to queue partitions 0 and 1, while queue partition 2 has no job data being written. Therefore, the number of available queue partitions K can be reduced to 2. In this way, new job monitoring data will continue to be written to partition 0 or 1, and the resources of partition 2 can be reserved for other needs.
[0049] As the number of jobs increases, the number of queue partitions is increased to accommodate the growing amount of job monitoring data. The number of queue relay interfaces is consistent with the number of queue partitions. If the number of jobs increases and the number of queue partitions expands, the number of queue relay interfaces and data analysis libraries deployed together will increase. Conversely, if the number of jobs decreases, the number of available queue partitions will decrease, concentrating job monitoring data on a few specific queue partitions to avoid resource waste. For example, if partition 5 is currently receiving job monitoring data, and other partitions are not sending monitoring data, and partition 5's monitoring data reception capacity is not at its maximum, then newly running jobs will prioritize sending monitoring data to partition 5. Components such as queue relay interfaces that consume and analyze other data can be shut down depending on whether historical job execution data is still needed.
[0050] Optionally, at the monitoring data acquisition end, determining the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job includes: firstly, determining the corresponding available queue partition identifier by performing an inverse operation based on the job identifier of the target job using a preset algorithm; then, determining the corresponding queue partition identifier based on the available queue partition identifier, wherein the available queue partition identifier corresponds to the queue partition identifier in the message queue.
[0051] The algorithm used is the remainder algorithm, which is the algorithm described above. For each partition whose number is determined by the remainder, the monitoring data is sent to that partition. For example, if the job ID is 3, the number of available queue partitions is 3, and the remainder is 0, then the monitoring data for job ID 3 will be sent to queue partition 0. This ensures the order of the monitoring data, with the earliest monitoring data always appearing at the top.
[0052] It should be noted that the identifier of an available queue partition may differ from the actual queue partition representation, but they have a corresponding relationship. An available queue partition does not mean that the queue partition is physically unavailable, but rather that it is no longer used for scheduling purposes to conserve resources. Furthermore, the number of available queue partitions does not necessarily start from partition number 0. An example is shown in Table 2, which is a table showing the correspondence between available queue partition identifiers and actual queue partition identifiers.
[0053] Table 2 shows the correspondence between available queue partition identifiers and actual queue partition identifiers.
[0054] Available queue partition identifier Queue partition identifier 0 2 1 1
[0055] After determining the queue partition identifier corresponding to the target job in step S201, step S202, determining the address of the corresponding data analysis library based on the queue partition identifier, includes: determining the corresponding queue relay interface identifier based on the queue partition identifier, wherein the queue relay interface corresponds one-to-one with the queue partition, and the queue relay interface is used to pull the corresponding job monitoring data from the queue partition and push it to the corresponding data analysis library; and using the address of the queue relay interface corresponding to the queue relay interface identifier as the address of the corresponding data analysis library, wherein the queue relay interface and the corresponding data analysis library are set on the same network node device and share the address of the network node device.
[0056] Since there is a one-to-one correspondence between the queue transfer interface and the queue partition, and the data analysis library corresponds to the queue transfer interface, the data analysis library corresponding to the target job can be determined based on the queue transfer interface identifier. This data analysis library stores the job monitoring data of the target job. Then, based on the address of the corresponding data analysis library, a request can be sent to that data analysis library to retrieve the job monitoring data of the target job.
[0057] When a request is sent from the monitoring data acquisition end, the queue partition where the monitoring data of the job is located is determined based on the job identifier, that is, the identifier of the queue transfer interface is obtained. Based on the queue transfer interface, it is determined which data analysis library on which machine is processing the monitoring data of the job, and the request is sent to that data analysis library.
[0058] The queue relay interface and the corresponding data analysis library are set on the same network node device and share the address of the network node device. Therefore, the address of the data analysis library can be directly used as the address of the queue relay interface corresponding to the target job. This not only avoids the problem of occupying communication network resources when the queue relay interface and the data analysis library are located on different network node devices, but also reduces the amount of data processing and simplifies the data processing process.
[0059] The network node device can be a physical device or a virtual device. As a node in network communication, functional modules within the same network node device, such as queue relay interfaces and corresponding data analysis libraries, do not require external network data transmission. Internal communication can occur within this network node device, avoiding the inefficiency and error-prone nature of using network communication for data interaction and transmission between different functional modules. In this embodiment, the network node device can be a machine node or a virtual container, such as a Docker container. A keep-alive mechanism can be configured within the network node device; that is, if the program within the network node device terminates abnormally, it will automatically restart and continue running, providing a degree of disaster recovery capability for the data analysis library to retrieve job monitoring data.
[0060] For example, the job ID can determine which specific partition the job monitoring data is sent to, i.e., the queue partition identifier. Since there is a one-to-one correspondence between the queue partition identifier and the queue relay interface ID, the machine addresses of the queue relay interface and the data analysis database can be determined. This information is maintained during the deployment of the queue relay interface and the data analysis database. An example is shown in Table 3, which is a table showing the correspondence between queue relay interface IDs and machine addresses.
[0061] Table 3. Correspondence between queue transfer interface IDs and machine addresses
[0062] Queue Transfer Interface ID Machine address IP 0 10.121.70.178 1 10.121.70.179
[0063] Optionally, in step S203, before sending a data query request to the data analysis library based on the address and pulling the job monitoring data of the target job from the data analysis library, the method further includes: pulling the job monitoring data of the corresponding job from the corresponding queue transfer interface through the data analysis library at a preset speed; and storing the pulled job monitoring data in the data analysis library in order.
[0064] The above-mentioned pull can be understood as acquisition. The specific process is that the puller actively requests data from the pullee according to a certain strategy, such as different frequencies. The pullee responds to the request and sends the requested data to the puller.
[0065] like Figure 3As shown, from a data flow perspective, before requesting the target job's job monitoring data from the data analysis library, the data analysis library needs to retrieve the target job's job monitoring data from the queue transfer interface and store it.
[0066] Optionally, before pulling the job monitoring data of the corresponding job from the corresponding queue transfer interface at a preset speed through the data analysis library, the method further includes: consuming the job monitoring data of the corresponding job stored in the corresponding queue partition through the queue transfer interface, wherein multiple queue transfer interfaces corresponding to multiple different queue partitions consume data from the corresponding queue partitions through the same consumer group; and caching the consumed job monitoring data in the cache of the queue transfer interface.
[0067] like Figure 3 As shown, the job monitoring data for the target job in the queue relay interface is pulled from the queue partition of the message queue. Before the data analysis library pulls the job monitoring data from the queue relay interface, the queue relay interface needs to consume the job monitoring data stored in the corresponding queue partition first.
[0068] It should be noted that when a queue relay interface consumes data from a queue partition, multiple queue relay interfaces corresponding to different queue partitions consume data from the corresponding queue partitions through the same consumer group. In this embodiment, all queue relay interfaces can use the same consumer group ID to read data and specify to consume a specific queue partition. Which queue partition is consumed is determined by the ID of the queue relay interface. The ID of the queue relay interface is fixed after deployment and is a unique identifier for the queue relay interface. It is determined according to the number of queue partitions in the message queue, starting from 0 and incrementing one by one, corresponding to each queue partition.
[0069] Optionally, before the data analysis library retrieves the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed, the method further includes: if the data analysis library does not retrieve data from the queue transfer interface within a first preset time, pausing the queue transfer interface from consuming job monitoring data from the corresponding queue partition; and restarting the queue transfer interface from consuming job monitoring data from the corresponding queue partition when the data analysis library resumes retrieving data from the queue transfer interface.
[0070] The queue relay interface has a caching mechanism. Before the data analysis library pulls data at a fixed rate, the queue relay interface consumes data from the queue partition and caches it in memory. This data is then provided directly to the data analysis library when it needs to retrieve data. This avoids waiting for the data analysis library and improves its efficiency in pulling job monitoring data. Furthermore, if the data analysis library experiences a program crash—that is, fails to pull data from the queue relay interface within a first preset time—the queue relay interface will stop consuming data until the data analysis library recovers. This ensures the stability of job monitoring data retrieval and avoids data loss due to excessive caching.
[0071] In this embodiment, the data analysis library also includes a corresponding alarm manager (Alertmanagent, a Prometheus alarm manager used for deduplication, classification, and degradation of alarm messages). Optionally, the method further includes: pushing alarm data from the job monitoring data to the alarm manager through the data analysis library, wherein the alarm data is the portion of the job monitoring data that needs alarm detection, and the alarm manager, queue relay interface, and corresponding data analysis library are located on the same network node device, sharing the address of the network node device; receiving alarm information pushed by the alarm manager, wherein the alarm manager is used to analyze the alarm data in the data analysis library according to preset rules and generate alarm information.
[0072] After the data analysis library retrieves the job monitoring data, the data that needs to be monitored and alarmed can be sent to the alarm manager. After receiving the data, the alarm manager can analyze the data according to preset rules, determine whether an alarm is needed, generate an alarm detection log, and generate alarm information when an alarm is needed, and actively push it to the monitoring data acquisition terminal, so that the system has the function of automatic alarm and detection.
[0073] In this scenario, in addition to the aforementioned queue relay interface and corresponding data analysis library, the alarm manager can also be located on the network node device. This not only avoids the problem of consuming communication network resources when the queue relay interface, data analysis library, and alarm manager are located on different network node devices, but also reduces data processing volume and simplifies the data processing flow. Furthermore, a keep-alive mechanism can still be configured on this network node device to ensure that the system has a certain degree of disaster recovery capability.
[0074] In some extreme cases, the network node devices may all crash, making it impossible to enable the keep-alive mechanism. In order to ensure the normal acquisition of job monitoring data, a heartbeat reporting mechanism is used to monitor the network node devices.
[0075] Optionally, if no heartbeat message is received from the queue relay interface within the second preset time, the network node device is determined to be down, and a new network node device is created based on the address of the network node device. In the new network node device, a new queue relay interface is created based on the queue relay interface identifier, as well as a new data analysis library and a new alarm manager corresponding to the new queue relay interface, which serve as backup components to consume job monitoring data from the queue partition.
[0076] Figure 4 This is a flowchart of a backup method for retrieving job monitoring data according to an embodiment of this application, such as... Figure 4 As shown, in the event of a system-wide outage, a new network node device can be deployed based on the outage address of the queue relay interface (i.e., the queue relay interface identifier mentioned above), and a new set of components can be created on the network node device. Furthermore, a new consumption strategy for the queue relay interface can be configured, such as starting data consumption from a specific historical moment and determining how much monitoring data to process to recover the job monitoring data lost due to the outage. The maximum recovery capability of the job monitoring data depends on the storage lifecycle of the job monitoring data in the queue partition.
[0077] Figure 5 This is a flowchart illustrating the process of retrieving job monitoring data from the monitoring data acquisition terminal provided in the embodiments of this application, such as... Figure 5 As shown, the monitoring data acquisition end first obtains the queue partition identifier based on the job ID; then, based on the queue partition identifier, it determines the addresses of the queue relay interface Kafka Reporter and the data analysis library Prometheus; based on these addresses, it sends a monitoring query request to the data analysis library Prometheus to retrieve the job monitoring data corresponding to the request. Meanwhile, the Alertmanager proactively sends the alert information to the monitored alarm information collection and storage program, so that the latest alarm information can be directly obtained when querying job monitoring information.
[0078] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0079] This application also provides a work monitoring data processing apparatus. It should be noted that this work monitoring data processing apparatus can be used to execute the work monitoring data processing method provided in this application. The work monitoring data processing apparatus provided in this application is described below.
[0080] Figure 6This is a schematic diagram of a work monitoring data processing device according to an embodiment of this application, such as... Figure 2 As shown, the device includes: a determination module 62, an address module 64, and a pull module 66. The device will be described in detail below.
[0081] The determination module 62 is used to determine the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job. The target job is the job for which monitoring data needs to be obtained, and the job monitoring data is stored in a unique corresponding data analysis library through the corresponding queue partition. The address module 64 is connected to the determination module 62 and is used to determine the address of the corresponding data analysis library based on the queue partition identifier. The retrieval module 66 is connected to the address module 64 and is used to send a data query request to the data analysis library based on the address to retrieve the job monitoring data of the target job from the data analysis library.
[0082] The job monitoring data processing device provided in this application pushes the job monitoring data of the job system to a message queue partition and then pulls it into a data analysis library. This achieves stable acquisition of job monitoring data. When a target job needs to be acquired, the device determines the address of the corresponding data analysis library based on the job identifier of the target job and accesses the data analysis library to obtain the corresponding job monitoring data. This achieves the goal of pulling job monitoring data on demand, improving the efficiency of job monitoring data acquisition, reducing the pressure on monitoring data acquisition, eliminating the need to continuously pull and store all job monitoring data, and correspondingly reducing the loss and errors of job monitoring data, thus lowering the failure rate of job monitoring data retrieval. This solves the problem in related technologies where, with the increase in the number of jobs, the monitoring data acquisition end directly pulls monitoring data from the processing engine, which puts a lot of pressure on the monitoring data acquisition end and causes a high failure rate.
[0083] The device for processing the job monitoring data includes a processor and a memory. The aforementioned determination module 62, address module 64, and retrieval module 66 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to achieve the corresponding functions.
[0084] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the problem of increased monitoring data and the inefficient method of directly pulling monitoring data from the processing engine in big data processing engines, which puts significant pressure on the data acquisition end and leads to a high failure rate, can be addressed.
[0085] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0086] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a method for processing job monitoring data.
[0087] This invention provides a processor for running a program, wherein the program executes a method for processing the job monitoring data during runtime.
[0088] Figure 7 This is a schematic diagram of an electronic device provided according to an embodiment of this application, such as... Figure 7 As shown, this application embodiment provides an electronic device 70, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above methods.
[0089] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.
[0090] This application also provides a computer program product that, when executed on a job monitoring data processing device, is suitable for executing a program that initializes any of the above-described method steps.
[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable job monitoring data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable job monitoring data processing apparatus, generate instructions for implementing the process... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.
[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable job monitoring data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0094] These computer program instructions can also be loaded onto a computer or other programmable device for processing monitoring data, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0095] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0096] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0097] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0098] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0100] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method of processing work monitoring data, characterized by, include: Based on the job identifier of the target job, determine the queue partition identifier corresponding to the job monitoring data of the target job, wherein the target job is the job for which monitoring data needs to be obtained, and the job monitoring data is stored in a unique corresponding data analysis library through the corresponding queue partition; Based on the queue partition identifier, determine the address of the corresponding data analysis database; Based on the address, a data query request is sent to the data analysis database to retrieve the job monitoring data of the target job from the data analysis database; Before sending a data query request to the data analysis library based on the address and retrieving the job monitoring data of the target job from the data analysis library, the method further includes: The data analysis library retrieves the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed; The retrieved job monitoring data is stored sequentially in the data analysis database; Before retrieving the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed through the data analysis library, the method further includes: The job monitoring data of the corresponding job stored in the corresponding queue partition is consumed through the queue transfer interface. Multiple queue transfer interfaces corresponding to multiple different queue partitions consume data from the corresponding queue partitions through the same consumer group. The consumed job monitoring data is cached in the cache of the queue transfer interface.
2. The method of claim 1, wherein, Based on the queue partition identifier, the address of the corresponding data analysis database is determined as follows: The corresponding queue transfer interface identifier is determined based on the queue partition identifier, wherein the queue transfer interface corresponds one-to-one with the queue partition, and the queue transfer interface is used to pull the corresponding job monitoring data from the queue partition and push it to the corresponding data analysis library; The address of the queue relay interface corresponding to the queue relay interface identifier is used as the address of the corresponding data analysis library. The queue relay interface and the corresponding data analysis library are deployed on the same network node device and share the address of the network node device.
3. The method of claim 1, wherein, Before retrieving the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed through the data analysis library, the method further includes: If the data analysis library fails to retrieve data from the queue transfer interface within a first preset time, the queue transfer interface will be suspended from consuming job monitoring data from the corresponding queue partition. If the data analysis library resumes pulling data from the queue transfer interface, restart the queue transfer interface to consume job monitoring data from the corresponding queue partition.
4. The method of claim 2, wherein, The method further includes: The alarm data in the job monitoring data is pushed to the alarm manager through the data analysis library. The alarm data is the part of the job monitoring data that needs to be detected. The alarm manager, the queue transfer interface and the corresponding data analysis library are set on the same network node device and share the address of the network node device. The system receives alarm information pushed by the alarm manager, wherein the alarm manager is used to analyze the alarm data in the data analysis library according to preset rules and generate the alarm information.
5. The method of claim 4, wherein, The method further includes: If no heartbeat message is received from the queue relay interface within a second preset time, it is determined that the network node device is completely down, and a new network node device is created based on the address of the network node device. In the new network node device, a new queue transfer interface is created based on the queue transfer interface identifier, along with a new data analysis library and a new alarm manager corresponding to the new queue transfer interface, which serve as backup components to consume the job monitoring data from the queue partition.
6. The method of claim 1, wherein, Before determining the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job, the method further includes: Determine the available queue partitions in the message queue, wherein the message queue includes multiple queue partitions, and the available queue partition is at least one partition selected from the multiple queue partitions according to a preset storage principle; When creating the target job, the job identifier of the target job is generated by a preset algorithm based on the available queue partition identifier of the available queue partition corresponding to the target job.
7. The method according to claim 6, characterized in that, Based on the job identifier of the target job, the queue partition identifier corresponding to the job monitoring data of the target job is determined as follows: Based on the job identifier of the target job, the corresponding available queue partition identifier is determined by performing an inverse operation using the preset algorithm; Based on the available queue partition identifier, the corresponding queue partition identifier is determined, wherein the available queue partition identifier corresponds to the queue partition identifier in the message queue.
8. A device for processing work monitoring data, characterized in that, include: The determination module is used to determine the queue partition identifier corresponding to the job monitoring data of the target job based on the job identifier of the target job, wherein the target job is the job for which monitoring data needs to be acquired, and the job monitoring data is stored in a unique corresponding data analysis library through the corresponding queue partition; The address module is used to determine the address of the corresponding data analysis library based on the queue partition identifier; The pull module is used to send a data query request to the data analysis library according to the address, and pull the job monitoring data of the target job from the data analysis library; Specifically, before sending a data query request to the data analysis library based on the address and retrieving the job monitoring data of the target job from the data analysis library, the retrieval module is further configured to: The data analysis library retrieves the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed; The retrieved job monitoring data is stored sequentially in the data analysis database; Before retrieving the corresponding job monitoring data from the corresponding queue transfer interface at a preset speed through the data analysis library, the retrieval module is also used for: The job monitoring data of the corresponding job stored in the corresponding queue partition is consumed through the queue transfer interface. Multiple queue transfer interfaces corresponding to multiple different queue partitions consume data from the corresponding queue partitions through the same consumer group. The consumed job monitoring data is cached in the cache of the queue transfer interface.
9. A computer-readable storage medium, characterized in that, The storage medium is used to store a program, wherein the program executes the method for processing job monitoring data according to any one of claims 1 to 7.
10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the job monitoring data processing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method and device for processing multi-cluster job record, equipment and storage medium
CN110795257A
