Method, device, equipment and storage medium for early warning of interruption of operation data

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product for early warning of job data interruption. The method can be used in the field of financial technology or other related fields, including: obtaining monitoring information corresponding to multiple job data, and obtaining data transmission identification information corresponding to multiple job data; obtaining data interruption monitoring results of each job data according to the monitoring information; obtaining target job data with data interruption prediction results indicating data interruption, and obtaining associated job data associated with the target job data from the job data according to the target data transmission identification information of the target job data; and providing data interruption risk early warning for the associated job data. This method can accurately provide early warning for batch job data interruption.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, device, computer equipment, storage medium and computer program product for early warning of job data interruption. Background Art

[0002] With the development of big data technology, batch job data monitoring technology has emerged. This technology monitors job data by monitoring each job data individually.

[0003] However, batch operation data are intricately intertwined. In the above technical solution, each operation data is monitored separately, the monitoring coverage is small, and there is a lack of prior interruption warning. An alarm is only issued after a batch interruption occurs, which makes the monitoring and warning results of batch operation data inaccurate. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for accurately warning of batch job data interruption in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for early warning of job data interruption. The method comprises:

[0006] Acquire monitoring information respectively corresponding to a plurality of operation data, and acquire data transmission identification information respectively corresponding to the plurality of operation data;

[0007] According to the monitoring information, obtaining data interruption monitoring results of each operation data;

[0008] Acquire the target operation data for which the data interruption prediction result indicates that there is data interruption, and acquire the associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data;

[0009] A data interruption risk warning is issued for the associated operation data.

[0010] In one of the embodiments, obtaining associated job data associated with the target job data from the job data based on the target data transmission identification information of the target job data includes: obtaining associated data transmission identification information associated with the target data transmission identification information and candidate job data corresponding to the associated data transmission identification information from the data transmission identification information; and obtaining the associated job data from the candidate job data.

[0011] In one of the embodiments, the data transmission identification information carries a position number; the position number is used to characterize the position information of each job data in the data chain constructed by the multiple job data; obtaining the associated job data from the candidate job data includes: obtaining the associated position number carried in the current associated data transmission identification information, and the target position number carried in the target data transmission identification information; when the difference between the associated position number and the target position number is less than a preset difference threshold, the candidate job data corresponding to the current associated data transmission identification information is used as the associated job data.

[0012] In one of the embodiments, the data interruption risk warning for the associated job data includes: obtaining an interruption risk warning level corresponding to the current associated job data based on the sequence number difference; and performing a data interruption risk warning for the current associated job data according to the interruption risk warning level.

[0013] In one of the embodiments, before obtaining the data transmission identification information respectively corresponding to the multiple job data, it also includes: obtaining a pre-obtained transmission protocol between the respective job data; the transmission protocol is used to characterize the data transmission direction between the respective job data; based on the transmission protocol, obtaining the data transmission identification information respectively corresponding to the multiple job data.

[0014] In one embodiment, the monitoring information includes multiple sub-monitoring information of different dimensions; based on the monitoring information, the data interruption monitoring result of each operation data is obtained, including: obtaining multiple sub-monitoring information corresponding to the current operation data; based on the multiple sub-monitoring information, obtaining multiple sub-monitoring results corresponding to the multiple sub-monitoring information respectively; based on the multiple sub-monitoring results, obtaining the data interruption monitoring result of the current operation data.

[0015] In one of the embodiments, obtaining the data interruption monitoring result of the current job data based on the multiple sub-monitoring results includes: obtaining the weight coefficient corresponding to each sub-monitoring result; and obtaining the data interruption monitoring result of the current job data based on each sub-monitoring result and the weight coefficient corresponding to each sub-monitoring result.

[0016] In one of the embodiments, after the data interruption risk warning is issued for the associated operation data, the method further includes: obtaining an interruption analysis result of the target operation data; obtaining an interruption emergency strategy corresponding to the associated operation data based on the interruption analysis result; and performing emergency processing on the associated operation data based on the interruption emergency strategy.

[0017] In a second aspect, the present application also provides a job data interruption warning device. The device comprises:

[0018] An identification information acquisition module, used to acquire monitoring information respectively corresponding to a plurality of operation data, and to acquire data transmission identification information respectively corresponding to the plurality of operation data;

[0019] A monitoring result acquisition module, used for acquiring data interruption monitoring results of each operation data according to the monitoring information;

[0020] an associated operation data acquisition module, configured to acquire target operation data for which the data interruption prediction result indicates that data interruption exists, and acquire associated operation data associated with the target operation data from the operation data according to target data transmission identification information of the target operation data;

[0021] The data interruption risk warning module is used to issue a data interruption risk warning to the associated operation data.

[0022] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0023] Acquire monitoring information respectively corresponding to a plurality of operation data, and acquire data transmission identification information respectively corresponding to the plurality of operation data;

[0024] According to the monitoring information, obtaining data interruption monitoring results of each operation data;

[0025] Acquire the target operation data for which the data interruption prediction result indicates that there is data interruption, and acquire the associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data;

[0026] A data interruption risk warning is issued for the associated operation data.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0028] Acquire monitoring information respectively corresponding to a plurality of operation data, and acquire data transmission identification information respectively corresponding to the plurality of operation data;

[0029] According to the monitoring information, obtaining data interruption monitoring results of each operation data;

[0030] Acquire the target operation data for which the data interruption prediction result indicates that there is data interruption, and acquire the associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data;

[0031] A data interruption risk warning is issued for the associated operation data.

[0032] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0033] Acquire monitoring information respectively corresponding to a plurality of operation data, and acquire data transmission identification information respectively corresponding to the plurality of operation data;

[0034] According to the monitoring information, obtaining data interruption monitoring results of each operation data;

[0035] Acquire the target operation data for which the data interruption prediction result indicates that there is data interruption, and acquire the associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data;

[0036] A data interruption risk warning is issued for the associated operation data.

[0037] The above-mentioned operation data interruption warning method, device, computer equipment, storage medium and computer program product obtain monitoring information corresponding to multiple operation data respectively, and obtain data transmission identification information corresponding to multiple operation data respectively; obtain data interruption monitoring results of each operation data according to the monitoring information; obtain target operation data with data interruption prediction results as having data interruption, obtain associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data; and issue data interruption risk warning for the associated operation data. The present application obtains data interruption monitoring results of operation data through monitoring information corresponding to batch operation data. If the data of the operation data is interrupted, a data interruption risk warning is issued for the associated operation data based on the operation data and the data transmission identification information of the data associated with the operation data. It is able to accurately issue warnings for batch operation data interruptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a process flow of a method for early warning of job data interruption in one embodiment;

[0039] Figure 2 A schematic diagram of a process for obtaining associated operation data in one embodiment;

[0040] Figure 3A schematic diagram of a process for obtaining associated operation data in another embodiment;

[0041] Figure 4 It is a schematic diagram of a process flow of operation data interruption risk warning in another embodiment;

[0042] Figure 5 A schematic diagram of a process flow of a full-link monitoring emergency system for operation data in one embodiment;

[0043] Figure 6 A schematic diagram of a process flow of batch diagnosis of job data in one embodiment;

[0044] Figure 7 It is a structural block diagram of a job data interruption warning device in one embodiment;

[0045] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be noted that the terms "first\second" involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0048] In one embodiment, Figure 1 As shown, a method for early warning of interruption of operation data is provided. This embodiment takes the method applied to a terminal as an example for explanation. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0049] Step S101, obtaining monitoring information respectively corresponding to a plurality of operation data, and obtaining data transmission identification information respectively corresponding to a plurality of operation data.

[0050] Among them, the job data refers to a large amount of job data, which is based on a single application or system as the basic granularity. The large amount of job data can build a corresponding data chain according to the direction of data transmission, and the monitoring information refers to the job data transmission and attribute information used to monitor whether the above job data is transmitted normally. For example, the monitoring information can be an indicator that indirectly reflects the risk of interruption, such as: CPU usage, memory occupancy, number of blocked threads, number of waiting threads on the application side; the current number of connections, current number of processes, table space usage, and file data size involved in the pending job used by the database used by the batch job. As for the data transmission identification information, it refers to the TAG tag marked by the data transmitted in the data link in the form of a tree node. The TAG tags of multiple data in the same transmission branch have the same root ID, and the TAG tags of multiple data in the same transmission branch have different position sequence IDs. The position sequence ID is used to characterize the position of multiple data in the same transmission branch in the transmission branch.

[0051] Specifically, the monitoring information respectively corresponding to the operation data and the data transmission identification information respectively corresponding to the operation data are acquired from the module where the operation data is located.

[0052] Step S102: acquiring data interruption monitoring results of each operation data according to the monitoring information.

[0053] The data interruption monitoring results include whether the operation data is interrupted or not.

[0054] Specifically, based on the monitoring information, the monitoring information is input into a data interruption monitoring model, and based on the data interruption monitoring model, the data interruption monitoring result of each operation data is obtained.

[0055] Step S103, obtaining target operation data for which the data interruption prediction result is that there is data interruption, and obtaining associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data.

[0056] Among them, the existence of data interruption means that the data interruption prediction result is the result of data interruption, the target operation data is the operation data of the data interrupted at the node where the data is located, and the associated operation data is the data on the same branch chain as the target operation data, or the associated operation data is the data that has a transmission relationship with the target operation data, that is, the associated operation data is the data that will be transmitted to the node where the target operation data is interrupted.

[0057] Specifically, if the data interruption prediction result of the current job data is that there is a data interruption, the current job data is used as the target job data, and according to the target data transmission identification information of the target job data, the associated job data corresponding to the data transmission identification information associated with the target data transmission identification information is found.

[0058] Step S104: issuing a data interruption risk warning for the associated operation data.

[0059] Among them, the data interruption risk warning is a warning of the possibility of data interruption in the operating data.

[0060] Specifically, according to the data transmission identification information of the associated operation data and the pre-defined data interruption risk warning rules, a data interruption risk warning is performed on the associated operation data.

[0061] The above-mentioned operation data interruption warning method obtains monitoring information corresponding to multiple operation data respectively, and obtains data transmission identification information corresponding to multiple operation data respectively; obtains data interruption monitoring results of each operation data according to the monitoring information; obtains target operation data with data interruption prediction results indicating data interruption, obtains associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data; and issues data interruption risk warning for the associated operation data. The present application obtains data interruption monitoring results of operation data through monitoring information corresponding to batch operation data. If the operation data is interrupted, data interruption risk warning is issued for the associated operation data based on the operation data and the data transmission identification information of the data associated with the operation data. It is able to accurately issue warnings for batch operation data interruptions.

[0062] In one embodiment, Figure 2 As shown, according to the target data transmission identification information of the target job data, the associated job data associated with the target job data is obtained from the job data, including the following steps:

[0063] Step S201: acquiring, from the data transmission identification information, associated data transmission identification information associated with the target data transmission identification information, and candidate job data corresponding to the associated data transmission identification information.

[0064] The associated data transmission identification information is data transmission identification information associated with the target data transmission identification information, and the associated data transmission identification information may be data transmission identification information having the same root ID as the target data transmission identification information. The candidate operation data is operation data on the same data transmission linear link as the target data.

[0065] Specifically, the root ID of the target data transmission identification information is obtained from the data transmission identification information, the data transmission identification information identical to the root ID of the target data transmission identification information is used as the associated data transmission identification information, and then the candidate job data corresponding to the associated data transmission identification information is further obtained.

[0066] Step S202: Acquire related job data from the candidate job data.

[0067] Specifically, the associated job data is acquired from the candidate job data according to the associated data transmission identification information of the candidate job data and the target data transmission identification information.

[0068] In this embodiment, by acquiring the associated data transmission identification information, the associated job data can be accurately acquired from the candidate job data.

[0069] In one embodiment, Figure 3 As shown, the data transmission identification information carries a position number; the position number is used to represent the position information of each operation data in the data chain constructed by multiple operation data; obtaining the associated operation data from the candidate operation data includes the following steps:

[0070] Step S301: Acquire the associated position number carried in the current associated data transmission identification information and the target position number carried in the target data transmission identification information.

[0071] Among them, the associated position serial number is the serial number of the job data carried in the associated data transmission identification information used to represent the candidate job data position in the data link, and the target position serial number is the serial number of the job data carried in the target data transmission identification information used to represent the target job data position in the data link.

[0072] Specifically, the associated position sequence number is obtained from the current associated data transmission identification information, and then the target position sequence number is obtained from the target data transmission identification information. For example, if the current associated data transmission identification information is A3, the associated position sequence number is 3, and the target data transmission identification information is A5, then the target position sequence number is 5, where A is the root ID in the data transmission identification information, which is used to characterize the data transmission chain where the job data is located.

[0073] Step S302: when the serial number difference between the associated position serial number and the target position serial number is less than a preset difference threshold, the candidate operation data corresponding to the current associated data transmission identification information is used as the associated operation data.

[0074] The sequence number difference is the difference between the associated position sequence number and the target position sequence number. The smaller the difference is, the closer the candidate job data is to the target job data, and the higher the risk of data interruption is. The difference threshold is a preset value set according to historical data.

[0075] Specifically, when the sequence number difference between the associated position sequence number and the target position sequence number is smaller than a preset difference threshold, the candidate operation data corresponding to the current associated data transmission identification information is used as the associated operation data.

[0076] In this embodiment, the associated operation data can be obtained quickly and accurately by calculating the difference between the associated position number and the target position number.

[0077] In one embodiment, Figure 4 As shown, the data interruption risk warning for the associated operation data includes the following steps:

[0078] Step S401, based on the sequence number difference, obtain the interruption risk warning level corresponding to the current associated operation data.

[0079] Among them, the interruption risk warning level is the urgency of the interruption risk warning of the associated operation data. The smaller the sequence number difference is, the higher the interruption risk warning level of the associated operation data is.

[0080] Specifically, based on a plurality of interruption risk warning levels pre-set based on historical data, the interruption risk warning level corresponding to the sequence number difference is found as the interruption risk warning level corresponding to the current associated job data.

[0081] Step S402: According to the interruption risk warning level, a data interruption risk warning is issued for the current associated operation data.

[0082] Specifically, according to the interruption risk warning level corresponding to the current associated operation data, the warning prompt information and the position of the current associated operation data in the data chain are visualized on the user side.

[0083] In this embodiment, by using the corresponding relationship between the interruption risk warning level and the sequence number difference, it is possible to timely and accurately issue a data interruption risk warning to the current associated operation data.

[0084] In one embodiment, before obtaining the data transmission identification information corresponding to multiple job data respectively, the following steps are also included: obtaining the transmission protocol between the pre-obtained job data; the transmission protocol is used to characterize the data transmission direction between the said each job data; based on the transmission protocol, obtaining the data transmission identification information corresponding to the multiple job data respectively.

[0085] The transmission protocol is the direction of data transmission between data. For example, "data A is transmitted to data B" is a transmission protocol.

[0086] Specifically, the transmission protocols between the various operation data obtained in advance are obtained, and according to these transmission protocols and predetermined rules, the data transmission identification information of each operation data can be obtained, and then the transmission data chain of these operation data can be constructed. For example, according to the predetermined data transmission identification information generation rule, the data No. 01 of a certain data packet generated by the A1 application and transmitted to B1 can be marked as A1-B1-01, and by analogy, the data No. 03 transmitted by B1 to C2 can be marked as A1-B1-01-C2-03.

[0087] In this embodiment, data transmission identification information respectively corresponding to a plurality of operation data can be efficiently obtained through the transmission protocol.

[0088] In one embodiment, the monitoring information includes a plurality of sub-monitoring information of different dimensions; obtaining the data interruption monitoring result of each operation data according to the monitoring information includes the following steps:

[0089] Acquire multiple sub-monitoring information corresponding to the current operation data; based on the multiple sub-monitoring information, obtain multiple sub-monitoring results corresponding to the multiple sub-monitoring information; and obtain the data interruption monitoring result of the current operation data according to the multiple sub-monitoring results.

[0090] Among them, multiple sub-monitoring information of different dimensions are different monitoring indicators. For example, multiple sub-monitoring information of different dimensions can be batch job start time, execution time, etc., application-side CPU usage, memory occupancy, number of blocked threads, number of waiting threads, current number of connections to the database used by the batch job, current number of processes, table space usage, file data size involved in the pending job, etc. The sub-monitoring result is the monitoring result corresponding to each sub-monitoring information.

[0091] Specifically, multiple sub-monitoring information is input into the data interruption monitoring model, and the sub-monitoring result corresponding to each sub-monitoring information is obtained through the data interruption monitoring model. The data interruption monitoring model further calculates the multiple sub-monitoring results obtained to obtain the data interruption monitoring result of the current operation data.

[0092] In this embodiment, the data interruption monitoring result of the current operation data can be obtained more accurately through multiple sub-monitoring information.

[0093] In one embodiment, obtaining a data interruption monitoring result of current operation data according to a plurality of sub-monitoring results includes the following steps:

[0094] Obtain the weight coefficient corresponding to each sub-monitoring result; based on each sub-monitoring result and the weight coefficient corresponding to each sub-monitoring result, obtain the data interruption monitoring result of the current operation data.

[0095] Among them, the weight coefficient is the weight of each sub-monitoring result to the final data interruption monitoring result.

[0096] Specifically, a corresponding weight coefficient is assigned to each sub-monitoring result, and the data interruption monitoring result of the current operation data is calculated.

[0097] In this embodiment, the data interruption monitoring result of the current operation data can be obtained more accurately through the weight coefficient corresponding to each sub-monitoring result.

[0098] In one embodiment, after the data interruption risk warning is issued for the associated operation data, the following steps are also included:

[0099] Obtain interruption analysis results of target operation data; obtain interruption emergency strategies corresponding to associated operation data based on the interruption analysis results; and perform emergency processing on associated operation data based on the interruption emergency strategies.

[0100] Among them, the interruption analysis result is the analysis result of the target operation data that has been interrupted, the interruption emergency strategy is the emergency strategy to prevent the related operation data from being interrupted, and the emergency treatment is a pre-set emergency treatment measure corresponding to the emergency strategy.

[0101] Specifically, through the interruption analysis model, the analysis results of the target operation data that has been interrupted are obtained. Based on the analysis results and the interruption risk warning level corresponding to the associated operation data, the corresponding pre-formulated emergency strategies and emergency handling measures are obtained, and emergency handling is performed on the associated operation data with interruption risks.

[0102] In this embodiment, by interrupting the emergency strategy, emergency processing can be performed on the associated operation data in a timely manner.

[0103] In an application embodiment, a batch job data full-link monitoring emergency system is provided, which can realize the linkage monitoring and disposal of upstream and downstream links, and link-based real-time monitoring of the execution status of each batch job in the batch link, the generation status of data files, and key operating indicators of the system and application sides such as CPU, memory, cluster capacity, job execution time, data file size, application upload logs, etc., to timely predict whether a batch interruption will occur in the short term or timely monitor whether a batch interruption has occurred. Once an abnormality is detected, timely diagnosis and automated emergency response are carried out to achieve rapid batch recovery, ensure the efficiency and quality of batch data operations, and effectively prevent irreparable losses and impacts. The batch job full-link monitoring emergency system is used to monitor batch job data files in real time, intelligently predict the possibility of interruption and automatically handle emergencies. The main process of the system is as follows:

[0104] The application uses tags to identify data transmitted in the entire link in the form of tree nodes. The database transmitted in the link has a unique root ID and a propagation ID for each node. The application accesses the file monitoring system and uploads logs in the specified format. The system monitors the application side, the system side, and the logs, and visualizes the monitoring of batch jobs in the link through the root ID. The application in the link configures custom emergency rules and plans for the interruption of various batch tasks. If the system detects and diagnoses the risk of interruption, it will immediately link an alarm and automatically implement the emergency plan to deal with the emergency. Manual emergency response can be performed in some complex scenarios.

[0105] Specifically, if Figure 5 As shown in the figure, first, the batch task of the starting application in the link starts to execute. The application side and the platform side send batch task data, batch data file information (including TAG ID), application logs, and system side data such as CPU usage, JVM, etc. to the monitoring module. The monitoring module will visualize the batch monitoring situation on the monitoring page, and at the same time call the diagnosis processing module to confirm whether the batch is interrupted, and make an estimated judgment on the interruption according to the distance classification of each node application from the abnormal application node. The diagnosis processing module will make a multi-dimensional prediction of the interruption possibility based on the monitoring information, and output a predicted interruption expectation and interruption cause. If the interruption cause exceeds the threshold, an alarm will be triggered and enter the emergency module. Once a batch interruption occurs, the emergency module will match the emergency rule library. If it matches the emergency rule pre-stored by the application, the relevant compensation transaction will be immediately started to ensure batch recovery. If the match fails or the emergency compensation fails, it will be thrown into the manual intervention process for manual emergency disposal until the batch is restored.

[0106] 1. The monitoring module is divided into two parts: one is the collection and monitoring of real-time relevant data and logs on the platform side and the application side; the other is the visualization of various monitoring indicators of batch jobs, including the following parts:

[0107] The data collection module supports pulling monitoring indicator data from various source servers and storing it in a time series database in a standardized format. Since this system focuses on monitoring batch data jobs and provides the ability to automatically warn and respond to batch interruptions, it is necessary to conduct a comprehensive collection of various direct and indirect indicators that can reflect the risk of batch interruptions. Among them, indicators that directly reflect the risk of interruption include: various log information output when the application side executes batch jobs; batch data size; batch job start time, execution time, etc. Indicators that indirectly reflect the risk of interruption include: CPU usage, memory occupancy, number of blocked threads, number of waiting threads on the application side; the current number of connections, current number of processes, table space usage, and file data size involved in the pending job of the database used by the batch job. The above-mentioned monitoring indicator collection sources may be logs, application batch server statistical information service interfaces, database statistical information service interfaces, etc. In the collection module, monitoring indicators from various sources are collected by the corresponding front-end collection adapters and stored in the time series database in a standardized format after formatting.

[0108] The monitoring and processing module can analyze various types of raw data stored in the time series database by the acquisition module, and such raw data are called raw indicators. At the same time, in order to provide more flexible and personalized monitoring needs, the module also supports users to perform composite processing on multiple raw indicators through online configuration to form new processing indicators. At the same time, the link monitors and warns potential hidden dangers of upstream and downstream nodes of the link according to the current batch running node application situation. An example is that by comparing the utilization rate of the CPU (or memory, etc.) with the number of free connection pools of the database (or pooled resources such as http), a processing indicator that can reflect the concurrent status of the batch job can be obtained. If the utilization rate of the pooled resources is low, but the server performance load is high, it means that the performance consumed by a single task is high, and you can consider horizontal expansion of the server to increase the number of concurrent connections to run full IO; if the utilization rate of the pooled resources is high, but the server performance load is low, it means that the performance consumed by a single task is low, and part of the server performance capacity can be recovered to avoid the waste of redundant resources.

[0109] The batch monitoring visualization module reads the original indicators and processing indicators of the monitoring processing module and displays them in a customized view. First, the CPU usage, memory usage, current number of processes, etc. of the current running node can be displayed in the form of a dashboard, and the size of batch data within a certain period of time can be displayed in the form of a line graph. Second, the relationship between upstream and downstream nodes is visualized in the form of a tree diagram.

[0110] 2. Batch diagnosis module: such as Figure 6As shown, this module uses a multi-dimensional interruption prediction model to analyze and predict interruption expectations and interruption causes. The establishment of the interruption prediction model can manually or automatically add interruption warning rules of different dimensions according to different job types, that is, interruption expectations and corresponding thresholds. The interruption expectation indicator can be a single-dimensional indicator, such as the CPU reaching 80% to trigger the interruption expectation; it can also be a combination of multiple dimensions (including processing indicators), such as the CPU reaching 70%, the hard disk usage rate reaching 90%, and then combined with other indicators to complete the setting. The historical interruption site data can be displayed to the manual setting personnel for reference, or after appropriate data intelligent adjustment, it can be automatically filled into the rules to avoid similar scene interruptions in the future.

[0111] Similar systems that are currently common usually need to issue an alarm only after a batch job is interrupted. This is a post-event alarm, or it is mainly aimed at the operation of a single application, and it is impossible to achieve link-based alarm. Downstream applications are not clear about the operation of upstream applications, and cannot make emergency preparations or compensation operations in advance. On this basis, this system establishes the association between upstream and downstream data transmission through the interruption prediction and diagnosis module, and performs analysis and prediction based on comprehensive link monitoring data. When there is a potential interruption in the system, an advance alarm is issued and transmitted to each application in the link, providing a basis for early intervention. Pre-event intervention can prevent batch interruptions before they occur, which can more effectively ensure the continuous and stable operation of the business and avoid unnecessary business impacts and public opinion risks. The interruption warning is also divided into two stages, one is the stage before the batch job starts, and the other is the stage when the batch job is in progress.

[0112] Before a batch job starts, you can collect dimensional data such as the type of batch job to be executed, the size of the file data that may be involved, and the amount of data that may be generated, to determine in advance whether the target server or database hard disk space is sufficient. If not, you can trigger the corresponding emergency plan in advance, including expanding the hard disk capacity, migrating files or backing up and migrating data, etc. In this way, when the batch job is actually executed, you will not encounter corresponding problems.

[0113] During the execution of batch jobs, the multi-dimensional interruption prediction model will pull the monitored system operation parameter indicators, such as CPU usage, memory usage, system load, swap usage percentage, network inbound packet rate, network outbound packet rate, disk usage, number of threads, number of processes, etc. The multi-dimensional interruption prediction model will analyze and match such parameters, calculate an interruption expectation, and perform intelligent matching based on the possible causes of the interruption. When the interruption expectation exceeds the set abnormal threshold, the system will determine that the batch task has an abnormal interruption risk, and give the most confident possible cause of the interruption, and immediately send an alarm email to the relevant personnel, and transfer it to the automated emergency module for emergency processing.

[0114] 3. The emergency module is divided into two parts: one is automated emergency, including application-customized batch emergency operations, compensation measures, and system-side emergency strategies; the other is manual emergency.

[0115] After the automated emergency module receives the (potential) interruption cause from the diagnostic module, it will give priority to matching the application's customized emergency rules or compensation measures in the emergency rule library. When the corresponding emergency measures are matched, the plan will be implemented for emergency response. For example, a batch interruption occurs in a job because a primary key conflict occurs in database table A when inserting new data. The application has been maintained in the customized emergency rule library in the early stage. When a primary key conflict occurs in table A, the date field will be compared, and the record with a newer time in the date field will be retained, and the older one will be discarded. The diagnostic module passes this interruption information and the interruption cause to the automated emergency module. The automated emergency module matches the rule from the customized rule library. At this time, according to the rule, the record with a newer time in the date field will be retained in table A, and the batch job will be restarted at the breakpoint.

[0116] If the application-defined emergency rules are not matched in the emergency rule library, the system will continue to match the pre-set general emergency rules. If the match is successful, the rules will be executed for emergency recovery. For example, if the batch interruption is caused by insufficient system memory, the system will execute elastic expansion commands to ensure the smooth execution of batch tasks.

[0117] If all the above emergency rules fail to match or the batch does not return to normal after executing the corresponding measures, an exception will be thrown to the manual emergency process, and manual emergency suggestions will be provided. After manual intervention by emergency personnel, further measures such as log analysis will be used to carry out emergency response until the batch tasks are restored.

[0118] In this embodiment, a full-link batch operation monitoring system capable of intelligent and automated emergency response is used to solve the problem that the existing batch operation system monitoring is mainly based on a single application (system) granularity and lacks linkage monitoring and disposal of the entire batch link. Combining the above advantages, this system can make batch operations run more safely, stably and efficiently, and also has more intelligent and automated operation and maintenance means to ensure the handling, processing, analysis and marketing of batch data, and promote the further development of big data and the digital economy.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a data interruption warning device for implementing the data interruption warning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data interruption warning device embodiments provided below can refer to the limitations of the data interruption warning method above, and will not be repeated here.

[0121] In one embodiment, Figure 7 As shown, a data interruption warning device is provided, including: an identification information acquisition module 701, a monitoring result acquisition module 702, a related operation data acquisition module 703, and a data interruption risk warning module 704, wherein:

[0122] The identification information acquisition module 701 is used to acquire monitoring information corresponding to a plurality of operation data, and acquire data transmission identification information corresponding to a plurality of operation data;

[0123] A monitoring result acquisition module 702 is used to acquire data interruption monitoring results of each operation data according to the monitoring information;

[0124] The associated operation data acquisition module 703 is used to acquire target operation data for which the data interruption prediction result is that there is data interruption, and acquire associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data;

[0125] The data interruption risk warning module 704 is used to issue a data interruption risk warning for the associated operation data.

[0126] In one embodiment, the associated job data acquisition module 703 is further used to acquire, from the data transmission identification information, associated data transmission identification information associated with the target data transmission identification information, and candidate job data corresponding to the associated data transmission identification information; and acquire the associated job data from the candidate job data.

[0127] In one of the embodiments, the associated job data acquisition module 703 is further used to obtain the associated position serial number carried in the current associated data transmission identification information and the target position serial number carried in the target data transmission identification information; when the serial number difference between the associated position serial number and the target position serial number is less than a preset difference threshold, the candidate job data corresponding to the current associated data transmission identification information is used as the associated job data.

[0128] In one embodiment, the data interruption risk warning module 704 is further used to obtain the interruption risk warning level corresponding to the current associated operation data based on the sequence number difference; and perform data interruption risk warning for the current associated operation data according to the interruption risk warning level.

[0129] In one embodiment, the identification information acquisition module 701 is further used to obtain the transmission protocol between each job data obtained in advance; the transmission protocol is used to characterize the data transmission direction between each job data; based on the transmission protocol, the data transmission identification information corresponding to multiple job data is obtained.

[0130] In one embodiment, the monitoring result acquisition module 702 is further used to obtain multiple sub-monitoring information corresponding to the current operation data; based on the multiple sub-monitoring information, multiple sub-monitoring results corresponding to the multiple sub-monitoring information are obtained; based on the multiple sub-monitoring results, the data interruption monitoring result of the current operation data is obtained.

[0131] In one embodiment, the monitoring result acquisition module 702 is further used to obtain the weight coefficient corresponding to each sub-monitoring result; based on each sub-monitoring result and the weight coefficient corresponding to each sub-monitoring result, the data interruption monitoring result of the current operation data is obtained.

[0132] In one embodiment, the data interruption risk warning module 704 is further used to obtain the interruption analysis results of the target operation data; obtain the interruption emergency strategy corresponding to the associated operation data according to the interruption analysis results; and perform emergency processing on the associated operation data based on the interruption emergency strategy.

[0133] Each module in the above-mentioned operation data interruption warning device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0134] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for early warning of job data interruption is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0135] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0138] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for early warning of operation data interruption, characterized in that: The method comprises: Acquire monitoring information corresponding to a plurality of job data respectively, and acquire data transmission identification information corresponding to the plurality of job data respectively; the monitoring information includes a plurality of monitoring indicators of different dimensions; the monitoring indicator is a direct indicator that directly reflects the interruption risk or an indirect indicator that indirectly reflects the interruption risk; the direct indicator is any one of various log information output when executing a batch job, the size of batch data, the start time of the batch job or the execution time of the batch job; the indirect indicator is any one of CPU usage rate, memory occupancy rate, number of blocked threads, number of waiting threads, the current number of connections of the database used by the batch job, the current number of processes of the database used by the batch job, the table space usage rate of the database used by the batch job or the size of file data involved in the pending jobs in the database used by the batch job; According to the monitoring information, data interruption monitoring results of the multiple job data are obtained, specifically including: pulling the multiple monitoring indicators through a multi-dimensional interruption prediction model and analyzing them to obtain interruption expectations; the multi-dimensional interruption prediction model is established based on interruption warning rules corresponding to different job types; when the interruption expectation exceeds a set abnormal threshold, determining that a batch task corresponding to the current job data has an abnormal interruption risk, and determining a possible cause of the interruption of the current job data; Acquire the target operation data for which the data interruption prediction result indicates that there is data interruption, and acquire the associated operation data associated with the target operation data from the operation data according to the target data transmission identification information of the target operation data; Performing a data interruption risk warning on the associated operation data, specifically comprising: visually displaying the warning prompt information and the position of the associated operation data in the data chain on the user end; Matching an automated emergency rule that matches the possible cause of the interruption in an emergency rule library, and executing the automated emergency rule; In the case that the operation data has not returned to normal, manual emergency suggestions are provided for possible causes of the interruption.

2. The method according to claim 1, characterized in that The acquiring, from the operation data, associated operation data associated with the target operation data according to the target data transmission identification information of the target operation data, comprises: Acquire, from the data transmission identification information, associated data transmission identification information associated with the target data transmission identification information, and candidate job data corresponding to the associated data transmission identification information; The associated job data is acquired from the candidate job data.

3. The method according to claim 2, characterized in that The data transmission identification information carries a position number; the position number is used to represent the position information of each of the operation data in the data chain constructed by the multiple operation data; The acquiring the associated job data from the candidate job data includes: Obtaining the associated position serial number carried in the current associated data transmission identification information and the target position serial number carried in the target data transmission identification information; When the sequence number difference between the associated position sequence number and the target position sequence number is less than a preset difference threshold, the candidate operation data corresponding to the current associated data transmission identification information is used as the associated operation data.

4. The method according to claim 3, characterized in that The performing of data interruption risk warning on the associated operation data includes: Based on the sequence number difference, obtaining the interruption risk warning level corresponding to the current associated operation data; According to the interruption risk warning level, a data interruption risk warning is performed on the current associated operation data.

5. The method according to claim 1, characterized in that Before acquiring the data transmission identification information respectively corresponding to the plurality of operation data, the method further includes: Acquire a transmission protocol between each of the operation data obtained in advance; the transmission protocol is used to characterize the data transmission direction between each of the operation data; Based on the transmission protocol, data transmission identification information respectively corresponding to the plurality of operation data is obtained.

6. The method according to claim 1, characterized in that The monitoring information includes a plurality of sub-monitoring information of different dimensions; and obtaining the data interruption monitoring results of the plurality of operation data according to the monitoring information includes: Get multiple sub-monitoring information corresponding to the current job data; Based on the multiple sub-monitoring information, obtain multiple sub-monitoring results corresponding to the multiple sub-monitoring information respectively; According to the multiple sub-monitoring results, the data interruption monitoring result of the current operation data is obtained.

7. The method according to claim 6, characterized in that The step of obtaining the data interruption monitoring result of the current operation data according to the plurality of sub-monitoring results includes: Obtaining weight coefficients corresponding to each of the sub-monitoring results; Based on each of the sub-monitoring results and the weight coefficient corresponding to each of the sub-monitoring results, the data interruption monitoring result of the current operation data is obtained.

8. The method according to claim 1, characterized in that After the data interruption risk warning is performed on the associated operation data, the method further includes: Obtaining an interruption analysis result of the target operation data; Obtaining an interruption emergency strategy corresponding to the associated operation data according to the interruption analysis result; Emergency processing is performed on the associated operation data based on the interruption emergency strategy.

9. A device for early warning of interruption of operation data, characterized in that: The device comprises: An identification information acquisition module is used to obtain monitoring information corresponding to multiple job data respectively, and to obtain data transmission identification information corresponding to the multiple job data respectively; the monitoring information includes multiple monitoring indicators of different dimensions; the monitoring indicator is a direct indicator that directly reflects the interruption risk or an indirect indicator that indirectly reflects the interruption risk; the direct indicator is any one of various types of log information output when executing batch jobs, batch data size, batch job start time or batch job execution time; the indirect indicator is any one of CPU usage rate, memory occupancy rate, number of blocked threads, number of waiting threads, current number of connections of the database used by the batch job, current number of processes of the database used by the batch job, table space usage rate of the database used by the batch job or file data size involved in pending jobs in the database used by the batch job; A monitoring result acquisition module is used to obtain data interruption monitoring results of the multiple job data according to the monitoring information, specifically including: extracting multiple monitoring indicators through a multi-dimensional interruption prediction model and analyzing them to obtain interruption expectations; the multi-dimensional interruption prediction model is established based on interruption warning rules corresponding to different job types; when the interruption expectation exceeds a set abnormal threshold, determining that the batch task corresponding to the current job data has an abnormal interruption risk, and determining the possible cause of the interruption of the current job data; an associated operation data acquisition module, configured to acquire target operation data for which the data interruption prediction result indicates that data interruption exists, and acquire associated operation data associated with the target operation data from the operation data according to target data transmission identification information of the target operation data; The data interruption risk warning module is used to issue a data interruption risk warning for the associated operation data, specifically including: visually displaying the warning prompt information and the position of the associated operation data in the data chain on the user side; matching the automated emergency rules that match the possible causes of the interruption in the emergency rule library, and executing the automated emergency rules; and providing manual emergency suggestions for the possible causes of the interruption when the operation data has not returned to normal.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images