Data monitoring method and device, computer equipment, storage medium and program product

Abstract

The invention discloses a data monitoring method and device, computer equipment, a storage medium and a program product, and relates to the technical field of data processing.The method comprises the steps that the current monitoring state of a target task is obtained; based on the monitoring stage of the monitoring state, determining operation data generated by the target task in the monitoring stage; if the running data triggers the data monitoring condition corresponding to the monitoring stage, determining a monitoring event type triggered by the target task; and performing data monitoring on the target task according to a monitoring strategy corresponding to the monitoring event type to generate a data monitoring result. By implementing the technical scheme of the invention, differentiated management can be carried out for different stages, fine management of monitoring is realized, and the timeliness and accuracy of data monitoring are greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a data monitoring method, apparatus, computer device, storage medium, and program product. Background Art

[0002] In the field of data quality, data quality check (DQC), as the mid-term defense line of the data quality system, is crucial for early detection of data quality problems and avoidance of major business impacts. Currently, the monitoring of data quality usually sets fixed monitoring thresholds based on experience and triggers an alarm when the data metrics exceed the thresholds. However, this method cannot consider the differences in different monitoring stages and the dynamic association between monitoring and business scenarios, making it difficult to ensure the timeliness and accuracy of data monitoring. Summary of the Invention

[0003] In view of this, the present disclosure provides a data monitoring method, apparatus, computer device, storage medium, and program product to solve the problem that it is difficult to ensure the timeliness and accuracy of data monitoring.

[0004] In a first aspect, the present disclosure provides a data monitoring method, including: obtaining the monitoring state in which a target task is currently located; determining the operation data generated by the target task in the monitoring stage based on the monitoring stage in which the monitoring state is located; if the operation data triggers the data monitoring condition corresponding to the monitoring stage, determining the type of monitoring event triggered by the target task; and performing data monitoring on the target task according to the monitoring strategy corresponding to the type of monitoring event to generate a data monitoring result.

[0005] In a second aspect, the present disclosure provides a data monitoring apparatus, including: a state obtaining module for obtaining the monitoring state in which a target task is currently located; an operation data determining module for determining the operation data generated by the target task in the monitoring stage based on the monitoring stage in which the monitoring state is located; an event type determining module for determining the type of monitoring event triggered by the target task if the operation data triggers the data monitoring condition corresponding to the monitoring stage; and a monitoring module for performing data monitoring on the target task according to the monitoring strategy corresponding to the type of monitoring event to generate a data monitoring result.

[0006] In a third aspect, the present disclosure provides a computer device, including: a memory and a processor, which are communicatively connected to each other, where the memory stores computer instructions, and the processor executes the computer instructions to execute the data monitoring method according to the first aspect or any corresponding implementation thereof.

[0007] Fourthly, the present disclosure provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the data monitoring method according to the first aspect or any corresponding embodiment thereof.

[0008] Fifthly, the present disclosure provides a computer program product, including computer instructions, and the computer instructions are used to cause a computer to execute the data monitoring method according to the first aspect or any corresponding embodiment thereof.

[0009] The data monitoring method, device, computer equipment, storage medium and program product provided by the present disclosure can determine the operation data generated by the target task in the monitoring stage by obtaining the current monitoring state of the target task and combining the monitoring stage where the monitoring state is located. Thus, the operation data corresponding to each monitoring stage can be collected, which is convenient for data quality monitoring in combination with each monitoring stage. When it is detected that the operation data triggers the data monitoring condition corresponding to the monitoring state, the type of monitoring event currently triggered by the target task is determined, so as to perform data monitoring on the target task according to the monitoring strategy corresponding to the type of monitoring event and generate a data monitoring result. Thus, corresponding monitoring can be carried out for monitoring events in different stages, so that differential management can be carried out for different stages, ensuring accurate monitoring of each monitoring node, realizing refined management of monitoring, effectively solving the problem of unclear management methods in different stages during the monitoring process, and being beneficial to greatly improving the timeliness and accuracy of data monitoring. Description of the Drawings

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 is a schematic diagram of the division of the monitoring stage according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flowchart of the data monitoring method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic overall diagram of data monitoring according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic flowchart of another data monitoring method according to an embodiment of the present disclosure;

[0015] Figure 5It is a schematic flowchart of another data monitoring method according to an embodiment of the present disclosure;

[0016] Figure 6 It is a schematic diagram of data monitoring for an admission event according to an embodiment of the present disclosure;

[0017] Figure 7 It is a schematic diagram of data monitoring for a departure event according to an embodiment of the present disclosure;

[0018] Figure 8 It is a schematic diagram of data monitoring for a weak-to-strong event according to an embodiment of the present disclosure;

[0019] Figure 9 It is a schematic diagram of data monitoring for a decommissioning event according to an embodiment of the present disclosure;

[0020] Figure 10 It is a block diagram of the structure of a data monitoring device according to an embodiment of the present disclosure;

[0021] Figure 11 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure. Detailed implementation manners

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0023] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, when responding to a user's active request, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operations of the technical solutions of the present disclosure according to the prompt message.

[0025] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving the user's active request may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above-mentioned notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0027] It is understandable that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0028] In the field of data quality, data quality check (DQC), as the mid-term defense line of the data quality system, is crucial for early detection of data quality problems and avoidance of major business impacts. In the actual operation and maintenance process, the deployment of a large number of DQC monitors brings many problems: First, the number of alarms is huge, and the energy of personnel is limited, so it is impossible to respond and process in time, resulting in delays in problems; Second, strong monitors alarm frequently, interfering with the normal output of data, making it difficult to stabilize the data output time and affecting business continuity; Third, the monitoring intensity of weak monitors is insufficient, and it is impossible to effectively block the problematic data link, resulting in the spread of data problems to the downstream and polluting more data; Fourth, when tasks are handed over or assets are taken offline, the monitors cannot be updated synchronously, making it difficult to ensure the timeliness and accuracy of monitoring.

[0029] The above problems seriously restrict the effectiveness of data quality monitoring and the exertion of data value. At present, the focus of data quality monitoring is mainly on the optimization of single monitoring indicators or the improvement of monitoring methods in specific scenarios. In related technologies, fixed monitoring thresholds are mainly set according to experience, and once the data indicators exceed the thresholds, alarms are triggered. However, this method lacks systematic management of the entire process of data quality monitoring, and does not fully consider the differences in different monitoring stages and the dynamic association between monitoring and business scenarios. For example, in terms of alarm handling, no classification processing is carried out according to the urgency of the alarm and the scope of business impact, resulting in important alarms being submerged in a large number of ordinary alarms; in the management of monitoring timeliness, no effective update mechanism is established for situations such as task handover and asset offline, making the monitoring out of touch with the actual business. At the same time, most of the existing data quality monitoring methods are static and difficult to adapt to the complex environment where the data scale and business requirements are constantly changing, and cannot fundamentally solve the current dilemmas faced by data quality monitoring.

[0030] Based on this, the technical solution of the present disclosure divides the life cycle of data quality monitoring into an observation period and an operation and maintenance period, and formulates differential monitoring strategies for different monitoring stages to achieve refined management of monitoring, as Figure 1 shown. At the same time, according to the real-time status and historical performance of the data, the monitoring configuration in different monitoring stages is automatically adjusted, as Figure 1The access, exit, weak-to-strong transition, offline, etc. shown break the limitations of traditional static monitoring and greatly improve the timeliness and accuracy of monitoring.

[0031] According to an embodiment of the present disclosure, an embodiment of a data monitoring method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0032] In this embodiment, a data monitoring method is provided, which can be used in computer devices such as computers, servers, etc. Figure 2 It is a flowchart of the data monitoring method according to an embodiment of the present disclosure, as Figure 2 shown, and the process includes the following steps:

[0033] Step S101, obtain the monitoring state where the target task is currently located.

[0034] The target task is a task for storing operation data in a data system. By successfully executing the target task, the smoothness of the operation data storage link in the data system is ensured, and then the continuity of the execution of the operation data storage service in the data system is ensured. The monitoring state is the state when monitoring the running metrics of the target task using a monitoring mechanism.

[0035] Specifically, as Figure 3 shown, a monitoring tool is deployed in the data system of the computer device. By detecting the running metadata of the target task during its operation using the monitoring tool, the corresponding running metrics are determined, and then by analyzing the running metrics, the monitoring state where the target task is currently located is determined.

[0036] Step S102, based on the monitoring stage where the monitoring state is located, determine the running data generated by the target task in the monitoring state.

[0037] The monitoring stage represents the stage where the target task is located in the monitoring life cycle. Specifically, the monitoring stage includes monitoring coverage, monitoring observation period, monitoring operation and maintenance period, and monitoring offline, etc. Among them, monitoring coverage means including the target task to be monitored; the monitoring observation period means observing the target task included in the monitoring; the monitoring operation and maintenance period means stably monitoring the target task; and monitoring offline means closing the monitoring of the target task.

[0038] The running data is the data generated by the target task in each monitoring stage, such as running success rate, number of alarms, etc. The running data of the target task can be characterized by the running metrics it has. Specifically, in each stage of monitoring the target task, the running data of the target task under each running metric can be collected by the monitoring tool.

[0039] In step S103, if the running data triggers the data monitoring condition corresponding to the monitoring phase, determine the type of monitoring event triggered by the target task.

[0040] The data monitoring condition is a pre-set condition for determining the type of monitoring event. Specifically, the user can set the corresponding data monitoring condition through the interactive component provided by the monitoring tool, such as the conditions that the number of alarms under different monitoring types needs to meet, the conditions that the running success rate under different monitoring types needs to meet, etc.

[0041] Different types of monitoring events have different data monitoring conditions, that is, there is a one-to-one correspondence between the monitoring event type and the data monitoring condition. By comparing the running data with the data monitoring condition, determine the data monitoring condition triggered by the running data. According to the data monitoring condition triggered by the running data, the type of monitoring event matching the data monitoring condition can be determined.

[0042] In step S104, perform data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type, and generate a data monitoring result.

[0043] The monitoring strategy is a strategy for monitoring and disposing of the target task, such as changing the monitoring intensity, adjusting the monitoring alarm method, taking the monitoring offline, etc. Since different types of monitoring events correspond to corresponding monitoring strategies, when the type of monitoring event is determined, the corresponding monitoring configuration can be updated using the type of monitoring event, and the monitoring strategy matching it can be determined. Subsequently, use this monitoring strategy to perform corresponding data monitoring on the operation of the target task to obtain the corresponding data monitoring result.

[0044] The data monitoring method provided in this embodiment can collect the running data corresponding to each monitoring phase by obtaining the current monitoring state of the target task and combining the monitoring phase in which the monitoring state is located, so as to facilitate data quality monitoring in combination with each monitoring phase. When it is detected that the running data triggers the data monitoring condition corresponding to the monitoring state, determine the type of monitoring event currently triggered by the target task, so as to perform data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type and generate a data monitoring result. Thus, corresponding monitoring can be performed on the monitoring events in different stages, so that differential management can be carried out for different stages, ensuring accurate monitoring of each monitoring node, realizing refined management of monitoring, effectively solving the problem of unclear management methods in different stages during the monitoring process, and being beneficial to greatly improving the timeliness and accuracy of data monitoring.

[0045] In this embodiment, a data monitoring method is provided, which can be used in computer devices such as computers and servers. Figure 4is a flowchart of a data monitoring method according to an embodiment of the present disclosure. As Figure 4 shown, the process includes the following steps:

[0046] Step S201, obtain the monitoring status of the target task at present. For details, please refer to the relevant description of the corresponding steps in the above - shown embodiment, which will not be elaborated here.

[0047] Step S202, based on the monitoring stage where the monitoring status is located, determine the operation data generated by the target task in the monitoring status. For details, please refer to the relevant description of the corresponding steps in the above - shown embodiment, which will not be elaborated here.

[0048] Step S203, if the operation data triggers the data monitoring condition corresponding to the monitoring stage, determine the type of monitoring event triggered by the target task.

[0049] Specifically, the above - mentioned step S203 includes:

[0050] Step S2031, when the target task is in the first monitoring stage, determine whether the operation data meets the first data monitoring condition in the first monitoring stage.

[0051] The first monitoring stage represents the state when the target task is in the monitoring observation period; the first data monitoring condition is the state switching condition set for the target task during the monitoring observation period. Compare the operation data generated by the target task in the first monitoring stage with the first data monitoring condition to determine whether the operation data meets the first data monitoring condition. If the operation data meets the first data monitoring condition, execute step S2032; otherwise, execute step S2033.

[0052] Step S2032, if the operation data meets the first data monitoring condition in the first monitoring stage, determine that the target task triggers the first monitoring event. Among them, the first monitoring event is an event that permits the transition from the first monitoring stage to the second monitoring stage, and the alarm level of the first monitoring stage is lower than that of the second monitoring stage.

[0053] The second monitoring stage represents the state when the target task is in the monitoring operation and maintenance period. When an abnormality occurs during the monitoring of the target task, its alarm level is higher than that of the first monitoring stage. Specifically, if the operation data of the target task meets the first data monitoring condition in its first monitoring stage, it is determined that the operation data of the target task meets the monitoring access standard, that is, the target task triggers an access event from the first monitoring stage to the second monitoring stage in the current monitoring status.

[0054] By analyzing the operation data during the monitoring observation period, when it is determined that the monitoring access conditions under the monitoring observation period are met, the monitoring of the target task is adjusted from the monitoring observation period to the monitoring operation and maintenance period, ensuring stable monitoring of the target task, achieving flexible changes in the monitoring stage, ensuring the dynamic association between the monitoring stage and the target task, facilitating adaptation to complex scenarios where the scale of operation data and task requirements change continuously, and ensuring the monitoring effect of data quality.

[0055] Step S2033, determine whether the operation data meets the second data monitoring condition under the first monitoring stage.

[0056] The second data monitoring condition is the monitoring shutdown condition set for the target task during the monitoring observation period. Compare the operation data generated by the target task under the first monitoring stage with the second data monitoring condition to determine whether the operation data meets the second data monitoring condition. If the operation data meets the second data monitoring condition, execute step S2034; otherwise, continue to monitor the operation data.

[0057] Step S2034, if the operation data meets the second data monitoring condition under the first monitoring stage, determine that the target task triggers the second monitoring event, where the second monitoring event indicates taking the target task offline.

[0058] If the operation data of the target task meets the second data monitoring condition under its first monitoring stage, it is determined that the operation data of the target task meets the monitoring offline standard, that is, the target task triggers a monitoring offline event in the current monitoring state.

[0059] In a specific example, create an alarm test group for the target task under the first monitoring stage and an alarm operation and maintenance group for the target task under the second monitoring stage; set the first data monitoring condition as: the number of effective alarm days in the recent X days (X = 7, etc.) <= 1, and the operation success rate in the recent X days (X = 7, etc.) is 100%; set the second data monitoring condition as: the number of effective alarm days in the recent X days (X = 30, etc.) >= 10 and the alarm response rate in the recent X days (X = 30, etc.) <= 50%, the operation success rate in the recent X days (X = 30, etc.) is 100%, and whether the task is offline = 1.

[0060] If, after analyzing the operation data, it is determined that the operation data simultaneously meets the number of effective alarm days and the operation success rate in the recent X days (X = 7, etc.), it can be determined that the target task triggers an access event to enter the second monitoring stage from the first monitoring stage. At this time, transfer the operation alarm information of the target task from the alarm test group to the alarm operation and maintenance group.

[0061] If, after analyzing the operation data, it is determined that the operation data meets the effective alarm days and alarm response rate in the recent X days (X = 30, etc.), or the operation success rate in the recent X days (X = 30, etc.) is 100%, or the task off - line status = 1, then the monitoring of the target task is closed to take the off - line processing of the monitoring of the target task.

[0062] By analyzing the operation data in the monitoring observation period, when it is determined that it meets the monitoring off - line conditions in the monitoring observation period, the monitoring of the target task is closed to save monitoring resources, so as to be able to take off - line the corresponding monitoring in time when the target task goes off - line, and avoid the disconnection between the monitoring and the target task.

[0063] Step S2035, when the target task is in the second monitoring stage, determine whether the operation data meets the third data monitoring conditions in the second monitoring stage.

[0064] The second monitoring stage represents the state when the target task is in the monitoring and operation and maintenance period; the third data monitoring conditions are the state transition conditions set for the target task in the monitoring and operation and maintenance period. Compare the operation data generated by the target task in the second monitoring stage with the third data monitoring conditions to determine whether the operation data meets the third data monitoring conditions. If the operation data meets the third data monitoring conditions, execute step S2036, otherwise execute step S2037.

[0065] Step S2036, if the operation data meets the third data monitoring conditions in the second monitoring stage, determine that the target task triggers the third monitoring event, and the third monitoring event is an event that permits the transition from the second monitoring state to the first monitoring state.

[0066] If the operation data of the target task meets its third data monitoring conditions in the second monitoring stage, it is determined that the operation data of the target task meets the monitoring exit standard, that is, the target task triggers an exit event from the second monitoring stage to the first monitoring stage in the current monitoring state.

[0067] By analyzing the operation data in the monitoring and operation and maintenance period, when it is determined that it meets the monitoring exit conditions in the monitoring and operation and maintenance period, adjust the monitoring of the target task from the monitoring and operation and maintenance period to the monitoring observation period, which can update the monitoring stage according to the operation situation of the target task, ensure the dynamic association between the monitoring stage and the target task, so as to select different monitoring strategies according to different monitoring stages and ensure the monitoring effect of data quality.

[0068] Step S2037, determine whether the operation data meets the fourth data monitoring conditions in the second monitoring stage.

[0069] The fourth data monitoring condition is the monitoring intensity switching condition set for the target task during the monitoring and operation and maintenance period. Compare the operation data generated by the target task in the second monitoring stage with the fourth data monitoring condition to determine whether the operation data meets the fourth data monitoring condition. If the operation data meets the fourth data monitoring condition, execute step S2038; otherwise, continue to monitor the operation data.

[0070] In step S2038, if the operation data meets the fourth data monitoring condition in the second monitoring stage, it is determined that the target task triggers the fourth monitoring event. Among them, the fourth monitoring event is the event of entering the second monitoring level from the first monitoring level, and the first monitoring level is lower than the second monitoring level.

[0071] The target task has corresponding monitoring levels and alarm levels in the second monitoring stage. The higher the monitoring level, the greater the monitoring intensity of the target task. Specifically, if the operation data of the target task meets the fourth data monitoring condition in the second monitoring stage, it is determined that the operation data of the target task meets the weak-to-strong monitoring standard, that is, the target task triggers a weak-to-strong event of entering the second monitoring level from the first monitoring level in the current monitoring state.

[0072] In a specific example, create an alarm test group for the target task in the first monitoring stage and an alarm operation and maintenance group for the target task in the second monitoring stage; set the third data monitoring condition as: the number of effective alarm days in the recent X days (X = 7, etc.) >= 2, and the operation success rate in the recent X days (X = 7, etc.) is not equal to 100%; set the fourth data monitoring condition as: the number of effective alarm days in the recent X days (X = 30, etc.) <= 1, the task operation duration in the recent X days (X = 30, etc.) <= 10 minutes, and the operation success rate in the recent X days (X = 30, etc.) is 100%.

[0073] If it is determined through analysis of the operation data that the operation data meets the number of effective alarm days in the recent X days (X = 7, etc.) >= 2, or meets the operation success rate in the recent X days (X = 7, etc.) is not equal to 100%, it can be determined that the target task triggers a quasi-exit event of entering the first monitoring stage from the second monitoring stage. At this time, the monitoring intensity of the target task in the second monitoring stage can be further obtained. If the target task is in the strong monitoring level in the second monitoring stage, adjust the monitoring level of the target task from the strong level to the weak level, adjust the alarm method to the alarm method of the weak level, and transfer the alarm information from the alarm operation and maintenance group to the alarm test group; if the target task is in the weak monitoring level in the second monitoring stage, directly transfer the alarm information from the alarm operation and maintenance group to the alarm test group.

[0074] If, after analyzing the operation data, it is determined that the operation data simultaneously meets the effective alarm days, task operation duration, and operation success rate in the recent X days (X = 30, etc.), then the monitoring level for the target task will be adjusted from the weak monitoring level to the strong monitoring level, and the alarm method will be adjusted to the alarm method of the strong level. For example, it will be adjusted from email alarm to phone alarm.

[0075] By analyzing the operation data in the monitoring and operation and maintenance period, when it is determined that it meets the monitoring weak-to-strong condition in the monitoring and operation and maintenance period, the monitoring level of the target task is adjusted. Thus, the automatic adjustment of the monitoring level according to the real-time state of the operation data is realized, the dynamic adjustment of the monitoring level is achieved, breaking the limit of traditional static monitoring, and greatly improving the timeliness and accuracy of monitoring. At the same time, the clear weak-to-strong standard prompts more important data to be more strictly monitored, improves the discovery and handling ability of key data quality problems, ensures the high-quality output of data, and improves the strong monitoring coverage rate.

[0076] Step S204, perform data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type, and generate a data monitoring result. For details, please refer to the relevant descriptions of the corresponding steps in the above-mentioned embodiments, and will not be elaborated here.

[0077] The data monitoring method provided in this embodiment divides the life cycle of data quality monitoring into a monitoring observation period and a monitoring and operation and maintenance period, and sets differentiated monitoring strategies for different monitoring stages to achieve refined management of monitoring, effectively solving the problem of unclear management methods in different monitoring stages during the monitoring process. Through the accurate conversion of the monitoring stage and the reasonable adjustment of the monitoring level, it is beneficial to filter out a large number of invalid alarms, reduce the alarm handling pressure, and make the alarm handling more targeted and efficient.

[0078] In this embodiment, a data monitoring method is provided, which can be used in computer devices such as computers and servers. Figure 5 is a flowchart of the data monitoring method according to an embodiment of the present disclosure, as Figure 5 shown, and this process includes the following steps:

[0079] Step S301, obtain the current monitoring state of the target task. For details, please refer to the relevant descriptions of the corresponding steps in the above-mentioned embodiments, and will not be elaborated here.

[0080] Step S302, based on the monitoring stage where the monitoring state is located, determine the operation data generated by the target task in the monitoring stage. For details, please refer to the relevant descriptions of the corresponding steps in the above-mentioned embodiments, and will not be elaborated here.

[0081] Step S303, obtain the monitoring indicators corresponding to the data monitoring conditions.

[0082] Monitoring metrics are used to characterize the metrics required for monitoring a target task, specifically including the effective number of alarms (such as the number of alarms, the number of alarm days), the operation success rate, the task operation duration, the alarm response rate, etc. Among them, the alarm volume represents the effective number of alarms triggered by monitoring. For example, when there are multiple alarms in a single day, it is counted as 1 day; the operation success rate represents the ratio of the number of successfully monitored instances to the total number of all monitored instances; the alarm response rate represents the ratio of the monitored alarm response volume to the total monitored alarm volume.

[0083] As described in the above embodiments, different data monitoring conditions have different monitoring metrics. Specifically, the monitoring tool provides an interactive page through which users can configure the monitoring metrics and obtain the monitoring metrics corresponding to each data monitoring condition.

[0084] Step S304: Extract the actual operation metrics of the target task in the monitoring state from the operation data.

[0085] The actual operation metrics are the metrics monitored during the actual operation process. The operation data of the target task during the actual operation process is collected through the monitoring tool and the operation data is analyzed. Then, according to the analysis result of the operation data, the actual operation metrics required for monitoring the target task are extracted from the operation data. Specifically, the data related to the execution of the target task can be collected from the task metadata, the data related to monitoring can be collected from the monitoring metadata, and the data related to operation and maintenance can be collected from the operation and maintenance metadata, as Figure 3 shown.

[0086] Step S305: Compare the actual operation metrics with the monitoring metrics, and determine whether the operation data triggers the data monitoring conditions corresponding to the monitoring state based on the result of the metric comparison.

[0087] Compare the actual operation metrics with the monitoring metrics to obtain the corresponding metric comparison result. Through this metric comparison result, determine whether the actual operation metrics of the target task in the monitoring state reach the monitoring metrics. If the metric comparison result indicates that the actual operation metrics of the target task in the monitoring state reach the monitoring metrics, it can be determined that the operation data triggers the data monitoring conditions corresponding to the monitoring state; if the metric comparison result indicates that the actual operation metrics of the target task in the monitoring state do not reach the monitoring metrics, it can be determined that the operation data does not trigger the data monitoring conditions corresponding to the monitoring state, and continue to monitor the operation data.

[0088] Step S306: If the operation data triggers the data monitoring conditions corresponding to the monitoring phase, determine the type of monitoring event triggered by the target task. For details, please refer to the relevant descriptions of the corresponding steps in the above embodiments, which will not be elaborated here.

[0089] Step S307: Monitor the data of the target task according to the monitoring policy corresponding to the monitoring event type, and generate a data monitoring result.

[0090] Specifically, the above step S307 includes:

[0091] Step S3071: According to the monitoring policy corresponding to the monitoring event type, adjust the monitoring level and / or alarm method for the target task to obtain the target monitoring level and / or target alarm method.

[0092] The monitoring level is used to represent the monitoring intensity for the target task; the alarm method is used to represent the method of abnormal alarm for the target task, such as telephone alarm, email alarm, SMS alarm, etc. Different types of monitoring events correspond to different monitoring policies, and different monitoring policies are configured with corresponding monitoring levels and alarm methods.

[0093] Specifically, there is a one-to-one correspondence between the monitoring event type and the monitoring policy. According to the correspondence between the monitoring event type and the monitoring policy, determine the monitoring policy corresponding to the monitoring event type. Then, according to the correspondence between the monitoring policy and the monitoring level, determine the target monitoring level corresponding to the monitoring event type. At the same time, according to the correspondence between the monitoring policy and the alarm method, the target alarm method corresponding to the monitoring event type can also be determined.

[0094] Step S3072: Monitor the running data of the target task according to the target monitoring level and / or target alarm method, and generate a data monitoring result.

[0095] The monitoring tool provides a monitoring configuration query interface and a monitoring configuration update interface. By calling the monitoring configuration query interface, the current monitoring configuration of the target task can be obtained. If the current monitoring configuration does not match the target monitoring level and / or target alarm method, then call the monitoring configuration update interface to adjust the monitoring level configured for the target task according to the target monitoring level, and at the same time, the alarm method configured for the target task can also be adjusted according to the target alarm method. Then, monitor the running data generated during the running process of the target task in real time according to the updated monitoring configuration, and generate the corresponding data monitoring result.

[0096] In some optional embodiments, the above method further includes:

[0097] Step S308: Record the data monitoring process corresponding to the data monitoring result to generate a data monitoring log, and store the data monitoring log in a preset location.

[0098] The preset location is a pre-set data storage location, such as configuring a corresponding database for each type of monitoring event. Specifically, the generation of data monitoring results has a corresponding data monitoring process, records the data monitoring process in the form of a log, obtains the data monitoring log generated by the target task at each monitoring stage of the monitoring life cycle, and stores the data monitoring log in the corresponding database.

[0099] In a specific example, for an admission event that meets the admission criteria, such as Figure 6 shown, according to the admission criteria, screen the target tasks, and determine the monitoring events that meet the admission data monitoring conditions. After the screening is completed, the current monitoring configuration of the target task can be obtained by calling the monitoring configuration query interface provided by the Data Quality Control (DQC) system, then call the corresponding monitoring configuration update interface, transfer the monitoring of the target task from the alarm test group to the alarm operation and maintenance group, record the current update log, and finally batch write it into the database.

[0100] In a specific example, for an exit event that meets the exit criteria, such as Figure 7 shown, according to the exit criteria, screen the target tasks. For the monitoring events that meet the exit data monitoring conditions, call the monitoring configuration update interface, and adjust the monitoring strength level, alarm method, alarm group, etc. respectively according to the strong and weak monitoring levels. At the same time, record the update log and write it into the database.

[0101] In a specific example, for a weak-to-strong event that meets the weak-to-strong criteria, such as Figure 8 shown, according to the weak-to-strong criteria, screen the target tasks, and determine the monitoring events that meet the weak-to-strong data monitoring conditions within the weak-to-strong range. For each monitoring event that meets the weak-to-strong conditions, call the monitoring configuration query interface provided by the Data Quality Control (DQC) system to obtain the current monitoring configuration of the target task, and call the monitoring configuration update interface to change the monitoring strength type and alarm method, record the update log and batch write it into the database.

[0102] In a specific example, for a decommissioning event that meets the decommissioning criteria, such as Figure 9 shown, according to the decommissioning criteria, screen the target tasks, determine the events that meet the decommissioning data monitoring conditions, call the monitoring batch shutdown interface provided by the Data Quality Control (DQC) system to complete the batch update operation, record all update log results and write them into the database.

[0103] The data monitoring method provided in this embodiment can determine the data monitoring conditions triggered by the current operation data according to the result of comparing the monitoring indicators with the actual operation indicators by comparing the monitoring indicators with the actual operation indicators. It is convenient to adjust the appropriate monitoring strategy according to the data monitoring conditions, and improve the monitoring quality. Determine the corresponding monitoring strategy in combination with the monitoring event type, and automatically update the monitoring configuration such as the monitoring level and the alarm method according to the monitoring strategy, without manual intervention, reduce human errors, improve the timeliness and accuracy of the monitoring configuration, ensure that the monitoring is always synchronized with the target task requirements, can reduce the delay of data output caused by data quality problems, improve the data service quality, and enhance the monitoring stability and reliability. Further, by recording the data monitoring logs generated during the data monitoring process, it is beneficial to query abnormal data according to the data monitoring logs.

[0104] In this embodiment, a data monitoring device is also provided. This device is used to implement the above-mentioned embodiment and the preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0105] This embodiment provides a data monitoring device, as Figure 10 shown, including:

[0106] A status acquisition module 401, configured to acquire the monitoring status of the target task at present.

[0107] An operation data determination module 402, configured to determine the operation data generated by the target task in the monitoring stage based on the monitoring stage where the monitoring status is located.

[0108] An event type determination module 403, configured to determine the type of monitoring event triggered by the target task if the operation data triggers the data monitoring condition corresponding to the monitoring stage.

[0109] A monitoring module 404, configured to perform data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type, and generate a data monitoring result.

[0110] In some alternative implementation manners, the event type determination module 403 includes:

[0111] A first judgment unit, configured to judge whether the operation data meets the first data monitoring condition in the first monitoring stage when the target task is in the first monitoring stage.

[0112] The first type determination unit is configured to determine that the target task triggers a first monitoring event if the operation data meets the first data monitoring condition in the first monitoring stage. The first monitoring event is an event that permits entering the second monitoring stage from the first monitoring stage, and the alarm level of the first monitoring stage is lower than that of the second monitoring stage.

[0113] In some alternative embodiments, the event type determination module 403 further includes:

[0114] The second judgment unit is configured to judge whether the operation data meets the second data monitoring condition in the first monitoring stage.

[0115] The second type determination unit is configured to determine that the target task triggers a second monitoring event if the operation data meets the second data monitoring condition in the first monitoring stage, where the second monitoring event indicates taking the target task offline.

[0116] In some alternative embodiments, the event type determination module 403 further includes:

[0117] The third judgment unit is configured to judge whether the operation data meets the third data monitoring condition in the second monitoring stage when the target task is in the second monitoring stage.

[0118] The third type determination unit is configured to determine that the target task triggers a third monitoring event if the operation data meets the third data monitoring condition in the second monitoring stage. The third monitoring event is an event that permits entering the first monitoring stage from the second monitoring stage.

[0119] In some alternative embodiments, the event type determination module 403 further includes:

[0120] The fourth judgment unit is configured to judge whether the operation data meets the fourth data monitoring condition in the second monitoring stage.

[0121] The fourth type determination unit is configured to determine that the target task triggers a fourth monitoring event if the operation data meets the fourth data monitoring condition in the second monitoring stage. The fourth monitoring event is an event of entering the second monitoring level from the first monitoring level, and the first monitoring level is lower than the second monitoring level.

[0122] In some alternative embodiments, the above device further includes:

[0123] The monitoring index acquisition module is configured to acquire the monitoring indexes corresponding to the data monitoring conditions.

[0124] The operation index extraction module is configured to extract the actual operation indexes of the target task in the monitoring state from the operation data.

[0125] An index comparison module is used to compare the actual operation indexes with the monitoring indexes, and determine whether the operation data triggers the data monitoring conditions corresponding to the monitoring status based on the index comparison result.

[0126] In some alternative embodiments, the monitoring module 404 includes:

[0127] An adjustment unit is used to adjust the monitoring level and / or alarm method for the target task according to the monitoring strategy corresponding to the monitoring event type, so as to obtain the target monitoring level and / or target alarm method.

[0128] An operation monitoring unit is used to monitor the operation data of the target task according to the target monitoring level and / or target alarm method, and generate a data monitoring result.

[0129] In some alternative embodiments, the above device further includes:

[0130] A record storage module is used to record the data monitoring process corresponding to the data monitoring result, generate a data monitoring log, and store the data monitoring log in a preset location.

[0131] The further function descriptions of the above modules and units are the same as those in the corresponding above embodiments, and will not be elaborated here.

[0132] The data monitoring device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0133] The data monitoring device provided in this embodiment can collect the operation data corresponding to each monitoring stage by obtaining the monitoring status of the target task currently, and combining the monitoring stage where the monitoring status is located, so as to determine the operation data generated by the target task in the monitoring stage, which is convenient for data quality monitoring in combination with each monitoring stage. When it is detected that the operation data triggers the data monitoring conditions corresponding to the monitoring status, the monitoring event type currently triggered by the target task is determined, so as to perform data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type and generate a data monitoring result. Thus, corresponding monitoring can be performed on the monitoring events in different stages, so that differential management can be carried out for different stages, ensuring accurate monitoring of each monitoring node, realizing refined management of monitoring, effectively solving the problem of unclear management methods in different stages during the monitoring process, and being beneficial to greatly improving the timeliness and accuracy of data monitoring.

[0134] The present disclosure embodiment also provides a computer device, having Figure 10The data monitoring device shown

[0135] Please refer to Figure 1 , Figure 11 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present disclosure. As Figure 11 shown, the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as a server array, a set of blade servers, or a multi-processor system). Figure 11 In

[0136]

[0137]

[0138]

[0139] Memory 20 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some optional embodiments, memory 20 may optionally include a memory remotely provided with respect to processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above networks include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid state drive; the memory 20 may further include a combination of the above types of memory.

[0140] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0141] Embodiments of the present disclosure also provide a computer-readable storage medium. The methods according to the embodiments of the present disclosure can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the methods described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state drive, etc.; further, the storage medium can also include a combination of the above types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0142] A part of the present disclosure can be applied as a computer program product, such as computer program instructions, which when executed by a computer, can call or provide the methods and / or technical solutions according to the present disclosure through the operation of the computer. Those skilled in the art should be able to understand that the forms of existence of computer program instructions in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0143] Although the embodiments of the present disclosure are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data monitoring method, characterized in that: The method comprises: Get the current monitoring status of the target task; Based on the monitoring stage of the monitoring state, determining the operation data generated by the target task in the monitoring stage; If the operation data triggers the data monitoring condition corresponding to the monitoring stage, determining the type of monitoring event triggered by the target task; Data monitoring is performed on the target task according to the monitoring strategy corresponding to the monitoring event type to generate a data monitoring result.

2. The method according to claim 1, characterized in that If the operation data triggers the data monitoring condition corresponding to the monitoring stage, determining the type of monitoring event triggered by the target task includes: When the target task is in the first monitoring stage, determining whether the operation data satisfies the first data monitoring condition in the first monitoring stage; If the operating data meets the first data monitoring condition under the first monitoring stage, it is determined that the target task triggers the first monitoring event. The first monitoring event is an event that allows entry from the first monitoring stage to the second monitoring stage. The alarm level of the first monitoring stage is lower than that of the second monitoring stage.

3. The method according to claim 2, characterized in that If the operation data triggers the data monitoring condition corresponding to the monitoring stage, determining the type of monitoring event triggered by the target task also includes: Determining whether the operating data satisfies a second data monitoring condition in the first monitoring stage; If the operating data meets the second data monitoring condition in the first monitoring stage, it is determined that the target task triggers a second monitoring event, and the second monitoring event represents that the target task is offline.

4. The method according to claim 2, characterized in that: If the operation data triggers the data monitoring condition corresponding to the monitoring stage, determining the type of monitoring event triggered by the target task also includes: When the target task is in the second monitoring stage, determining whether the operation data satisfies a third data monitoring condition in the second monitoring stage; If the operating data meets the third data monitoring condition in the second monitoring stage, it is determined that the target task triggers a third monitoring event, and the third monitoring event is an event that allows entry from the second monitoring state to the first monitoring state.

5. The method according to claim 4, characterized in that If the operation data triggers the data monitoring condition corresponding to the monitoring stage, determining the type of monitoring event triggered by the target task also includes: Determining whether the operating data satisfies a fourth data monitoring condition in the second monitoring stage; If the operating data meets the fourth data monitoring condition under the second monitoring stage, it is determined that the target task triggers a fourth monitoring event, and the fourth monitoring event is an event of entering the second monitoring level from the first monitoring level, and the first monitoring level is lower than the second monitoring level.

6. The method according to any one of claims 1 to 5, characterized in that: Determining whether the operation data triggers the data monitoring condition corresponding to the monitoring stage includes: Obtaining monitoring indicators corresponding to the data monitoring conditions; Extracting the actual operation index of the target task under the monitoring state from the operation data; The actual operation index is compared with the monitoring index, and based on the index comparison result, it is determined whether the operation data triggers the data monitoring condition corresponding to the monitoring stage.

7. The method according to any one of claims 1 to 5, characterized in that: The step of performing data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type and generating a data monitoring result includes: According to the monitoring strategy corresponding to the monitoring event type, the monitoring level and / or the alarm mode for the target task are adjusted to obtain the target monitoring level and / or the target alarm mode; According to the target monitoring level and / or target alarm mode, the operating data of the target task is monitored to generate the data monitoring result.

8. The method according to claim 1, characterized in that Also includes: The data monitoring process corresponding to the data monitoring result is recorded, a data monitoring log is generated, and the data monitoring log is stored in a preset location.

9. A data monitoring device, characterized in that: The device comprises: The status acquisition module is used to obtain the current monitoring status of the target task; An operation data determination module, used to determine the operation data generated by the target task in the monitoring stage based on the monitoring stage of the monitoring state; An event type determination module, configured to determine the type of monitoring event triggered by the target task if the operation data triggers the data monitoring condition corresponding to the monitoring stage; The monitoring module is used to perform data monitoring on the target task according to the monitoring strategy corresponding to the monitoring event type and generate data monitoring results.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data monitoring method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data monitoring method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the data monitoring method according to any one of claims 1 to 8.

Images