Job log monitoring method and device, computer device and storage medium

Abstract

The application relates to a job log monitoring method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining to-be-analyzed job log data of a job log center; the to-be-analyzed job log data is data obtained by processing log data generated by a target job during running; inputting the to-be-analyzed job log data into a log analysis model to obtain a job log data analysis result, judging whether the target job is interrupted based on the job log data analysis result to obtain a first judgment result; the log analysis model is a multi-layer structure bidirectional gate recurrent unit network; if the first judgment result is that the target job is not interrupted, inputting the to-be-analyzed job log data into an alarm tracking model to obtain a second judgment result corresponding to the job log data analysis result of the job not interrupted; if the second judgment result is an abnormal alarm, outputting abnormal running data for the target job. The method can improve monitoring efficiency and reduce the error recognition rate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, and storage medium for monitoring job logs. Background Technology

[0002] With the development of artificial intelligence technology, cloud technology has emerged. This technology can unify a series of resources such as hardware, software, and networks within a wide area network or local area network to realize a managed technology for data computing, storage, processing, and sharing.

[0003] In traditional technologies, manual and semi-automatic operation and maintenance methods are increasingly unable to meet the demands as data volumes grow. Current log monitoring and analysis of large volumes of logs largely rely on passive and time-consuming methods such as monitoring error reports and keyword searches, resulting in low monitoring efficiency and a high error rate. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for monitoring job logs based on attention mechanisms and neural networks that can provide abnormal alarms, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a method for monitoring job logs. The method includes: acquiring job log data to be analyzed from a job log center; the job log data to be analyzed is data obtained after processing log data generated by a target job during runtime; inputting the job log data to be analyzed into a log analysis model to obtain job log data analysis results; determining whether the target job is interrupted based on the job log data analysis results to obtain a first determination result; the log analysis model is a multi-layered bidirectional gated cyclic unit network; if the first determination result indicates that the target job is not interrupted, then inputting the job log data to be analyzed into an alarm tracking model to obtain a second determination result corresponding to the analysis result of the uninterrupted job log data; if the second determination result is an abnormal alarm, then outputting abnormal operation data for the target job.

[0006] In one embodiment, the step of inputting the job log data to be analyzed into a log analysis model to obtain job log data analysis results, and determining whether the job log data analysis results are interrupted to obtain a first determination result, includes: obtaining the log data record corresponding to the job name of the target job, the start time and end time corresponding to the log data record; inputting the log data record corresponding to the job name, the start time and end time corresponding to the log data record into the log analysis model to obtain the job log data analysis results; and determining whether the target job is interrupted based on the job log data analysis results to obtain the first determination result.

[0007] In one embodiment, the step of inputting the log data record corresponding to the job name, the start time and end time corresponding to the log data record into the log analysis model to obtain the job log data analysis result includes: calculating the job success rate and historical problem repair duration for the time period corresponding to the job name based on the log data record corresponding to the job name; obtaining the runtime corresponding to the log data record based on the start time and end time corresponding to the log data record; and inputting the job success rate, the historical problem repair duration and the runtime corresponding to the log data record into the log analysis model to obtain the job log data analysis result.

[0008] In one embodiment, if the first judgment result indicates that the target operation is not interrupted, then inputting the operation log data to be analyzed into the alarm tracking model to obtain a second judgment result corresponding to the analysis result of the uninterrupted operation log data includes: obtaining the table name or file name corresponding to the operation log data to be analyzed, wherein the table name or file name reflects the carrying status of the operation log data to be analyzed; inputting the table name or file name corresponding to the operation log data to be analyzed into the alarm tracking model to obtain a second judgment result corresponding to the analysis result of the uninterrupted operation log data; the second judgment result is used to characterize whether there is an abnormal alarm in the target operation.

[0009] In one embodiment, the step of inputting the table name or file name corresponding to the job log data to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data includes: obtaining the data volume corresponding to the job log data to be analyzed based on the table name or file name of the job log data to be analyzed; and obtaining the second judgment result corresponding to the analysis result of the uninterrupted job log data based on the start time, the runtime, the data volume corresponding to the job log data to be analyzed, and the environmental information input into the abnormal state calculation formula.

[0010] In one embodiment, the method further includes: inputting the job log data to be analyzed output by the target job corresponding to the abnormal alarm into a tracking algorithm model to obtain a third judgment result; if the third judgment result is abnormal, updating the running job data corresponding to the abnormal running information table; the abnormal running information table is used to record the abnormal information corresponding to the target job; if the third judgment result is normal, repeating the step of inputting the job log data to be analyzed output by the target job corresponding to the abnormal alarm into the tracking algorithm model, and recording the number of times the step is repeated.

[0011] In one embodiment, the step of repeatedly executing the target job corresponding to the abnormal alarm and inputting the job log data to be analyzed into the tracking algorithm model to obtain a fourth judgment result includes: if the number of times is less than a preset number threshold, then the job log data to be analyzed is again judged as abnormal for the target job; if the number of times is greater than or equal to the number threshold, then the abnormal monitoring status corresponding to the target job is lifted and normal job information is output.

[0012] Secondly, this application also provides a job log monitoring device. The device includes: a job log data acquisition module for acquiring job log data to be analyzed from a job log center; the job log data to be analyzed is data obtained after processing log data generated by a target job during runtime; a first judgment result acquisition module for inputting the job log data to be analyzed into a log analysis model to obtain a job log data analysis result, and judging whether the target job is interrupted based on the job log data analysis result to obtain a first judgment result; the log analysis model is a multi-layered bidirectional gated cyclic unit network; and an anomaly alarm module for inputting the job log data to be analyzed into an alarm tracking model if the first judgment result indicates that the target job is not interrupted, to obtain a second judgment result corresponding to the analysis result of the uninterrupted job log data; and for outputting abnormal operation data for the target job if the second judgment result is an anomaly alarm.

[0013] In one embodiment, the first judgment result obtaining module is used to obtain the log data record corresponding to the job name corresponding to the target job, the start time and end time corresponding to the log data record; input the log data record corresponding to the job name, the start time and end time corresponding to the log data record into the log analysis model to obtain the job log data analysis result; and judge whether the target job is interrupted based on the job log data analysis result to obtain the first judgment result.

[0014] In one embodiment, the first judgment result obtaining module is used to calculate the job success rate and historical problem repair duration for the time period corresponding to the job name based on the log data record corresponding to the job name; obtain the runtime corresponding to the log data record based on the start time and end time corresponding to the log data record; and input the job success rate, the historical problem repair duration and the runtime corresponding to the log data record into the log analysis model to obtain the job log data analysis result.

[0015] In one embodiment, the second judgment result obtaining module is used to obtain the table name or file name corresponding to the job log data to be analyzed, the table name or file name reflecting the carrying status of the job log data to be analyzed; input the table name or file name corresponding to the job log data to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data; the second judgment result is used to characterize whether there is an abnormal alarm in the target job.

[0016] In one embodiment, the second judgment result obtaining module is used to obtain the data volume corresponding to the job log data to be analyzed based on the table name or file name of the job log data to be analyzed; and to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data based on the start time, the runtime, the data volume corresponding to the job log data to be analyzed, and the environmental information input into the abnormal state calculation formula.

[0017] In one embodiment, the tracking module is used to input the job log data to be analyzed output by the target job corresponding to the abnormal alarm into the tracking algorithm model to obtain a third judgment result; if the third judgment result is abnormal, the corresponding running job data in the abnormal running information table is updated; the abnormal running information table is used to record the abnormal information corresponding to the target job; if the third judgment result is normal, the step of inputting the job log data to be analyzed output by the target job corresponding to the abnormal alarm into the tracking algorithm model is repeated, and the number of times the step is repeated is recorded.

[0018] In one embodiment, the tracking module is configured to, if the number of occurrences is less than a preset threshold, again output the task log data to be analyzed for the target task as abnormal; if the number of occurrences is greater than or equal to the threshold, then remove the abnormal monitoring status corresponding to the target task and output normal task information.

[0019] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring job log data to be analyzed from a job log center; the job log data to be analyzed is data obtained after processing log data generated by a target job during runtime; inputting the job log data to be analyzed into a log analysis model to obtain job log data analysis results; judging whether the target job is interrupted based on the job log data analysis results to obtain a first judgment result; the log analysis model is a multi-layered bidirectional gated cyclic unit network; if the first judgment result indicates that the target job is not interrupted, then inputting the job log data to be analyzed into an alarm tracking model to obtain a second judgment result corresponding to the analysis result of the uninterrupted job log data; if the second judgment result is an abnormal alarm, then outputting abnormal operation data for the target job.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps: acquiring job log data to be analyzed from a job log center; the job log data to be analyzed is data obtained after processing log data generated by a target job during runtime; inputting the job log data to be analyzed into a log analysis model to obtain job log data analysis results; determining whether the target job is interrupted based on the job log data analysis results to obtain a first determination result; the log analysis model is a multi-layered bidirectional gated cyclic unit network; if the first determination result indicates that the target job is not interrupted, then inputting the job log data to be analyzed into an alarm tracking model to obtain a second determination result corresponding to the analysis result of the uninterrupted job log data; if the second determination result is an abnormal alarm, then outputting abnormal operation data for the target job.

[0021] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps: acquiring job log data to be analyzed from a job log center; the job log data to be analyzed is data obtained after processing log data generated by a target job during runtime; inputting the job log data to be analyzed into a log analysis model to obtain job log data analysis results; determining whether the target job is interrupted based on the job log data analysis results to obtain a first determination result; the log analysis model is a multi-layered bidirectional gated cyclic unit network; if the first determination result indicates that the target job is not interrupted, then inputting the job log data to be analyzed into an alarm tracking model to obtain a second determination result corresponding to the analysis result of the uninterrupted job log data; if the second determination result is an abnormal alarm, then outputting abnormal operation data for the target job.

[0022] The aforementioned job log monitoring method, device, computer equipment, storage medium, and computer program product acquire job log data to be analyzed from a job log center. This job log data is the processed log data generated by the target job during runtime. The job log data is input into a log analysis model to obtain job log data analysis results. Based on these results, a first judgment result is obtained to determine whether the target job is interrupted. The log analysis model is a multi-layered bidirectional gated cyclic unit network. If the first judgment result indicates that the target job is not interrupted, the job log data is input into an alarm tracking model to obtain a second judgment result corresponding to the uninterrupted job log data analysis result. If the second judgment result is an abnormal alarm, abnormal operation data is output for the target job.

[0023] By acquiring the task log data to be analyzed from the task log center, the system can accurately and quickly obtain the data for analysis from the terminal, enabling a high degree of cooperation between the task log center and this analysis system and improving the stability between the two systems. Through the analysis of the log analysis model, the task log data analysis results are obtained, and a first judgment result is derived based on these results. This allows for multi-step analysis of the task log data to be analyzed, resulting in reliable analysis results. Moreover, the same analysis process is applied to each data point, ensuring consistency. The judgment result derived from the analysis results also continues these characteristics, helping to improve the efficiency and accuracy of the analysis of the task log data. By performing a second judgment on the task log data to be analyzed that was not interrupted in the first judgment, a judgment result of normal operation or abnormal alarm is obtained. This allows for a more careful classification of the task log data to be analyzed that did not cause system interruption, preventing defective data from flowing into the next step, further improving the judgment accuracy and efficiency of the system corresponding to this method.

[0024] By analyzing the task log data to be analyzed in the task log center, corresponding judgment results are obtained. It can perform real-time or near real-time analysis of TB-level logs, classify tasks and organize operation information. Based on attention mechanism and neural network, the intelligent log monitoring system effectively reduces the cost of manual analysis and monitoring, which addresses the problems of real-time log monitoring and intelligent analysis and processing in existing operation and maintenance system technologies. Attached Figure Description

[0025] Figure 1 This is an application environment diagram of a job log monitoring method in one embodiment;

[0026] Figure 2 This is a flowchart illustrating a job log monitoring method in one embodiment;

[0027] Figure 3 This is a flowchart illustrating a job log monitoring method in another embodiment;

[0028] Figure 4 This is a flowchart illustrating a job log monitoring method in yet another embodiment;

[0029] Figure 5 This is a flowchart illustrating a job log monitoring and alarm method in one embodiment;

[0030] Figure 6 This is a flowchart illustrating a job log monitoring and alarm method in another embodiment;

[0031] Figure 7 This is a flowchart illustrating a job log monitoring and tracking method in one embodiment;

[0032] Figure 8 This is a flowchart illustrating a job log monitoring and tracking method in another embodiment;

[0033] Figure 9 This is a schematic diagram of a deep bidirectional GRU network model based on an attention mechanism in one embodiment;

[0034] Figure 10 Here is a logic diagram of an alarm model in one embodiment;

[0035] Figure 11 Here is a logic diagram of the tracing model in one embodiment;

[0036] Figure 12 This is a flowchart of an intelligent log monitoring system based on attention mechanism and neural network in one embodiment;

[0037] Figure 13This is a structural block diagram of a job log monitoring device in one embodiment;

[0038] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] The job log monitoring method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 acquires data, server 104 receives the data from terminal 102 in response to the terminal 102's instructions, performs calculations on the acquired data, and transmits the calculation results back to terminal 102 for display. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed in the cloud or on other network servers. Server 104 obtains the job log data to be analyzed from the job log center at terminal 102. The job log data to be analyzed is the data obtained after processing the log data generated by the target job during runtime. The job log data to be analyzed is input into the log analysis model to obtain the job log data analysis result. Based on the job log data analysis result, a judgment is made on whether the target job is interrupted, resulting in a first judgment result. The log analysis model is a multi-layered bidirectional gated cyclic unit network. If the first judgment result indicates that the target job is not interrupted, the job log data to be analyzed is input into the alarm tracking model to obtain a second judgment result corresponding to the uninterrupted job log data analysis result. If the second judgment result is an abnormal alarm, abnormal operation data is output for the target job. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0041] In one embodiment, such as Figure 2 As shown, a method for monitoring job logs is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0042] Step 202: Obtain the job log data to be analyzed from the job log center.

[0043] The job log center can be a system that runs target jobs. This system can generate data related to job logs. By analyzing this data, the specific situation of target jobs of different natures in the job log center can be understood, and corresponding adjustments can be made to the job log center.

[0044] The task log data to be analyzed can be the data obtained after processing the log data generated by the target task during runtime. The task log data to be analyzed includes various types of data, such as log data records, table names, start time, and end time. The processing of the task log data to be analyzed obtained from the target task can be done by using a neural network with an attention mechanism to extract key information.

[0045] Specifically, the server responds to the terminal's instructions and retrieves unanalyzed job log data corresponding to at least one log center from the terminal. This unanalyzed job log data is then processed by an attention-based neural network to extract key information, resulting in the job log data to be analyzed. The terminal's retrieval of the job log data corresponding to the log center can be real-time or near real-time, with a near real-time delay not exceeding one minute. The entire retrieval process can involve extracting key information from a large number of job execution logs in the log center using an attention-based neural network. This includes retrieving the job start time, end time (including normal completion and error-related interruption times), the table name or filename of the data source, the generated filename or the table name to which it was written, the names of dependent preceding jobs, error keywords or log fragments (empty if the job completed without errors), server availability, and database availability. After retrieving the data, the server stores it in its storage unit. When data is needed, the server retrieves the corresponding data from the storage unit and processes it in the server's central processing unit.

[0046] For example, job log center A generates unanalyzed job log data corresponding to target job B. After the terminal detects the unanalyzed job log data, it inputs the data into the server. Then, through a neural network with a pre-trained attention mechanism in the server, key information in the log is extracted to obtain the job log data C to be analyzed. Data that needs to be calculated is directly input into volatile memory, while data that does not need to be processed temporarily is written to non-volatile memory for storage.

[0047] Step 204: Input the job log data to be analyzed into the log analysis model to obtain the job log data analysis results. Based on the job log data analysis results, determine whether the target job has been interrupted and obtain the first judgment result.

[0048] The log analysis model can be an artificial intelligence model (a multi-layered bidirectional gated recurrent unit network) used to analyze the log data of the job to be analyzed. The artificial intelligence model mentioned here can be an analytical artificial intelligence model, a functional artificial intelligence model, an interactive artificial intelligence model, a textual artificial intelligence model, or a visual artificial intelligence model. The server intelligently selects one type of artificial intelligence model according to the type of data and the target task requirements, or it can be selected through manual intervention.

[0049] The results of job log data analysis can be obtained by analyzing the job log data to be analyzed through an artificial intelligence model. The results of job log data analysis include various data, such as runtime, file size, file transcoding, and transmission duration.

[0050] The first judgment result can be the result output by the server's judgment unit on the analysis result of the job log data. The first judgment result can be a normal job, an abnormal job, or an interrupted job, etc. The judgment result in the judgment unit can be adjusted according to the actual situation.

[0051] Specifically, the log data of the job to be analyzed is input into the log analysis model. This model searches the database for records based on the job name, calculates the job success rate and the duration of historical problem repairs for a recent time period, obtains the runtime using the start and end times (interruption points), and determines the amount of data processed based on the table or file name of the data source. The generated file name provides the file size and the duration of file transcoding and transmission. Then, based on the job execution data processed in the first two steps, a judgment is made to obtain the first judgment result. First, it is determined whether the job was interrupted, and it is initially labeled as interrupted. Then, based on the start duration, runtime, data size, and environmental information (server availability, previous job execution status), the abnormal status calculation formula is used to label the job as normal or not interrupted but with abnormal execution information. Jobs labeled as normal are classified as normal jobs, while those that were interrupted or not interrupted but with abnormal execution information are classified as abnormal jobs.

[0052] This is a deep bidirectional GRU network model based on an attention mechanism. The model consists of an input layer, a DBGRU layer (Attention Mechanism in DBGRU part), a fully connected layer (FC layer part), and an output layer. The DBGRU layer is composed of three interleaved DBGRU layers and three interleaved attention layers, with 128 neurons per layer. The fully connected layer consists of two fully connected networks: the first has 64 neurons, and the second has 32 neurons. The job log is input into the model. After passing through the DBGRU layer and the fully connected layer, the output includes an error flag (0 - normal, 1 - error), an error cause analysis, and a proposed solution. A schematic diagram of the deep bidirectional GRU network model based on the attention mechanism is shown below. Figure 9 As shown.

[0053] The training process is as follows:

[0054] Historical information stored in the GRU network is subject to update gate z. (t) and reset door control r (t) x is the input data, and h is the cell output state of the GRU.

[0055] z (t) =σ(W xz x (t) +W hz h (t-1) +b z )

[0056] Where: z (t) It updates the current state value of the gate, z. (t) Subject to the current input data x (t) The output h of the GRU at the previous moment (t-1) The impact, W xz W hz These are the corresponding update gate weights, b z It is a bias parameter. The update gate controls the degree to which the state information from the previous moment is incorporated into the current state. The larger the value of the update gate, the more state information from the previous moment is incorporated.

[0057] r (t) =σ(W xr x (t) +W xr h (t-1) +b r )

[0058] Where: r (t) It resets the current state value of the door, z (t) Subject to the current input data x (t)The output h of the GRU at the previous moment (t-1) The impact, W xr W hr These are the corresponding reset gate weights, b r This is the bias parameter. The reset gate controls the degree to which the state information from the previous moment is ignored; the smaller the reset gate value, the more information is ignored.

[0059]

[0060] in: It is the hidden state value of the memory unit at the current moment. Subject to the current input data x (t) and the current reset gate r (t) Compared to the previous time step, the output h of the GRU (t-1) The effect of the product, These are the corresponding weights. It is a bias parameter.

[0061]

[0062] Where: h (t) It is the output value of the GRU unit at time t.

[0063] The gating mechanism of GRU allows the model to capture long-range historical information. To simultaneously acquire contextual information, a bidirectional GRU is used. Therefore, the output value h in BGRU... (t) It can be represented as follows:

[0064]

[0065] in: These are the forward and backward output values ​​of the input data in the GRU at time t, respectively. This indicates an integration operation.

[0066] Using all the above formulas, the output value h of all BGRU units can be obtained. (t) , will h (t) Input attention mechanism model. The attention mechanism used in this patent is the Soft Attention model, with the following formula:

[0067]

[0068] Where: C i (t) It is the output value of the i-th memory unit at time t after passing through the attention mechanism, a i (t) h is the attention allocation coefficient of the i-th memory unit at time t. i (t)This is the output value of the GRU unit at time t. At this point, the first-layer BGRU network and attention mechanism model obtain the first batch of parameters, and all C values ​​are... i (t) Input the second-layer BGRU network and repeat the above process until the three-layer BGRU and each layer's attention mechanism model obtain the first batch of parameters. Use the output of the last layer's attention mechanism model as the input to the fully connected network to start training the fully connected network, and finally set the loss function to mean absolute error.

[0069] For example, the task log data C to be analyzed is input into a pre-trained log analysis model for analysis. The log analysis model used here is a multi-layered bidirectional gated recurrent unit network, which yields the task log data analysis result D. Based on the log data analysis result D, the result D is input into the judgment unit in the server for type judgment, resulting in the first judgment result O.

[0070] Step 206: If the first judgment result is that the target operation is not interrupted, the log data of the operation to be analyzed is input into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted operation log data; if the second judgment result is an abnormal alarm, abnormal operation data is output for the target operation.

[0071] The alarm tracking model can be a model that performs a further judgment on the work log data to be analyzed, where the initial judgment result is that it is uninterrupted, to obtain the final judgment result. The alarm tracking model can employ artificial intelligence or ordinary judgment logic; the specific choice depends on the specific format of the work log data to be analyzed.

[0072] The second judgment result can be the result obtained by further judging the work log data to be analyzed without interruption of the first judgment result. The second judgment result mainly includes normal operation and abnormal alarm. New judgment aspects can also be introduced according to special needs, and the judgment results in the judgment unit can be adjusted according to the actual situation.

[0073] Among them, abnormal operation data can be the job log data to be analyzed corresponding to the target job whose second judgment result is an abnormal alarm. This data can reflect the specific situation of the abnormal alarm. If needed, it can also be input into the visualization terminal for visualization display.

[0074] Specifically, the log data of the target job to be analyzed, which is determined by the first judgment result to be uninterrupted, is input into the alarm tracking model for further judgment. This judgment yields a second judgment result corresponding to the uninterrupted job log data analysis result. The second judgment result is generally set as normal operation or abnormal alarm; however, it can also be a target category other than these two categories as needed. If the second judgment result is an abnormal alarm, the server outputs the log data of the target job to be analyzed as abnormal operation data. For the alarm tracking model, a judgment is first made. If there is no interruption or abnormality, it may be a misjudgment by the log analysis model. In this case, the job information is first organized and written into the normal operation information table, and then an emergency contact person is notified by email to analyze whether the job is normal. If the job is abnormal, the recent operation information of the job is first retrieved from the database, and the current abnormal information is added and sent to the visualization module for display. Simultaneously, an emergency contact person is automatically contacted by email to inform them of the job abnormality. Finally, the job information is written into the abnormal operation information table. The formula for calculating the abnormal status is as follows:

[0075]

[0076] Where S i This represents the i-th feature in the latest running information of job S. ji This represents the i-th feature information in the j-th recent run information of job S. This represents the average of the i-th feature information from the recent n runs of job S. δ is a manually set threshold; exceeding the threshold indicates an anomaly, while falling below it indicates normal operation. The logic diagram of the alarm algorithm model is shown below. Figure 10 As shown.

[0077] For example, if the first judgment result O is uninterrupted, the job log data C to be analyzed is input into the alarm tracking model for the next judgment to obtain the second judgment result P corresponding to the uninterrupted job log data analysis result. If the second judgment result P is normal operation, the normal operation information (including job start time, runtime, data size, dependencies, etc.) is recorded in the normal operation information table of the database according to the normal operation operation. If the second judgment result P is abnormal alarm, the job log data to be analyzed corresponding to the target job is output as abnormal operation data.

[0078] In the above-mentioned job log monitoring method, the job log data to be analyzed is obtained from the job log center; the job log data to be analyzed is the data obtained after processing the log data generated by the target job during runtime; the job log data to be analyzed is input into the log analysis model to obtain the job log data analysis result; based on the job log data analysis result, it is determined whether the target job is interrupted to obtain a first judgment result; the log analysis model is a multi-layer bidirectional gated cyclic unit network; if the first judgment result is that the target job is not interrupted, the job log data to be analyzed is input into the alarm tracking model to obtain a second judgment result corresponding to the uninterrupted job log data analysis result; if the second judgment result is an abnormal alarm, abnormal operation data is output for the target job.

[0079] By acquiring the task log data to be analyzed from the task log center, the system can accurately and quickly obtain the data for analysis from the terminal, enabling a high degree of cooperation between the task log center and this analysis system and improving the stability between the two systems. Through the analysis of the log analysis model, the task log data analysis results are obtained, and a first judgment result is derived based on these results. This allows for multi-step analysis of the task log data to be analyzed, resulting in reliable analysis results. Moreover, the same analysis process is applied to each data point, ensuring consistency. The judgment result derived from the analysis results also continues these characteristics, helping to improve the efficiency and accuracy of the analysis of the task log data. By performing a second judgment on the task log data to be analyzed that was not interrupted in the first judgment, a judgment result of normal operation or abnormal alarm is obtained. This allows for a more careful classification of the task log data to be analyzed that did not cause system interruption, preventing defective data from flowing into the next step, further improving the judgment accuracy and efficiency of the system corresponding to this method.

[0080] By analyzing the task log data to be analyzed in the task log center, corresponding judgment results are obtained. It can perform real-time or near real-time analysis of TB-level logs, classify tasks and organize operation information. Based on attention mechanism and neural network, the intelligent log monitoring system effectively reduces the cost of manual analysis and monitoring, which addresses the problems of real-time log monitoring and intelligent analysis and processing in existing operation and maintenance system technologies.

[0081] In one embodiment, such as Figure 3 As shown, the task log data to be analyzed is input into the log analysis model to obtain the task log data analysis results. A first judgment result is obtained based on whether the task log data analysis results are interrupted, including:

[0082] Step 302: Obtain the log data record corresponding to the job name of the target job, the start time of the log data record, and the end time of the log data record.

[0083] The log data records can be records of log data generation and retrieval in the server's storage unit, and these records include at least one message about log data changes.

[0084] The start time can be the time when the task corresponding to the target job begins to be processed. For example, the time when the server starts processing the target job is recorded as t1.

[0085] The end time can be the time when the task corresponding to the target job finishes processing. For example, the time recorded by the server when the target job finishes processing is t2.

[0086] Specifically, the server responds to the terminal's instructions, retrieves the log data record corresponding to the job name of the target job, the start time of the log data record, and the end time of the log data record from the terminal, and processes the above three data accordingly. For example, log data record 1 is associated with the start time, and log data record n ​​is associated with the end time. After the server obtains the data, it stores the data in the server's storage unit. When the data is needed, the server will retrieve the corresponding data from the storage unit and send it to the server's central processing unit for calculation.

[0087] For example, the server obtains three pieces of data corresponding to the job name of the target job from the terminal: log data record H, and the start time t1 and end time t2 corresponding to the log data record.

[0088] Step 304: Input the log data record corresponding to the job name, the start time and end time corresponding to the log data record into the log analysis model to obtain the job log data analysis results.

[0089] Specifically, the log data records corresponding to the job names, as well as the start and end times of these log data records (all three data points are from the log data of the job to be analyzed), are input into the log analysis model. Based on the job names, records are searched in the database to calculate the job success rate and the duration of historical problem repairs for a recent time period; the runtime is obtained using the start and end times (interruption points); the amount of data processed is determined based on the table name or file name of the data source; and the file size and file transcoding and transmission time are obtained based on the generated file name. The analysis results corresponding to the job log data are then obtained.

[0090] For example, the log data record H corresponding to the job name, as well as the start time t1 and end time t2 corresponding to the log data record H, are input into the log analysis model for calculation. The calculation results yield data such as file size, file transcoding, and transmission duration. The collection of these data is the job log data analysis result.

[0091] Step 306: Based on the analysis results of the job log data, determine whether the target job has been interrupted and obtain the first judgment result.

[0092] Specifically, the results of the job log data analysis are input into the judgment module of the log analysis model for judgment. This involves determining the amount of data to be processed (log data record, corresponding start and end times) based on the table or file name of the data source, and obtaining the file size and transcoding / transmission duration from the generated file name. Then, the job execution data is used for judgment to obtain the first judgment result. First, it checks if the job was interrupted; if so, it is labeled as interrupted. If the first judgment result is not an interruption, the process proceeds to the next step.

[0093] For example, the analysis result D of the job log data is input into the judgment function module in the log analysis model for judgment, and the first judgment result O is obtained. The first judgment result O mainly includes whether the operation is interrupted or not. If it is an interrupted operation, the calculation is stopped and the job log data C to be analyzed that caused the interrupted operation is output. If it is not an interrupted operation, the job log data C to be analyzed corresponding to this judgment is input into the next step.

[0094] In this embodiment, the operation log data analysis results are obtained by inputting log data records, start time, and end time into the log analysis model, and a first judgment result is further obtained. This can determine the influencing factors of the operation log data analysis results and the first judgment result, and the parameters of the operation log center cannot be corrected based on the first judgment result.

[0095] In one embodiment, such as Figure 4 As shown, the log data record corresponding to the job name, the start time and end time corresponding to the log data record are input into the log analysis model to obtain the job log data analysis results, including:

[0096] Step 402: Based on the log data records corresponding to the job name, calculate the job success rate and historical problem repair duration for the time period corresponding to the job name.

[0097] The job success rate can be the success rate of the target job in the log job center. This success rate can be obtained by analyzing and calculating the log data of the job to be analyzed. Only when the success rate is greater than the preset success rate is the target job in the log job center considered to be a qualified job.

[0098] Among them, the historical problem repair duration can be the length of time corresponding to the repair of the hardware or software of the log job center when a problem occurs. Since all situations of the log job center are recorded, the historical problem repair duration is also recorded in the job log data to be analyzed.

[0099] Specifically, based on the log data records corresponding to the job names, the job success rate and historical problem repair duration for the recent job names within the corresponding time periods are calculated. The job success rate is calculated by counting all job items and successful job items within the recent job name's corresponding time period, then dividing the number of successful job items by the total number of job items and multiplying by the percentage. The historical problem repair duration is calculated by subtracting the repair time from the fault initiation time in the job log center.

[0100] For example, based on the log data record H corresponding to the job name, all job items M and successful job items m are read. The initial fault time is t3 and the repair time is t4. The calculated job success rate is (m / M)×100%, and the calculated historical problem repair time is T2=t4-t3.

[0101] Step 404: Based on the start time and end time of the log data record, obtain the runtime corresponding to the log data record.

[0102] The runtime can be the difference between the start time and the end time of the log data record generated by the target job. Since the log data record corresponds to the target job, it is also the runtime of the target job.

[0103] Specifically, the runtime corresponding to the log data record is obtained by calculating the difference between the start time and end time in the log data record.

[0104] For example, if the start time in the log data is t1 and the end time is t2, the difference between these two time points is T1 = t2 - t1, which gives the runtime corresponding to the log data record.

[0105] Step 406: Input the job success rate, the duration of historical problem repairs, and the runtime corresponding to the log data records into the log analysis model to obtain the job log data analysis results.

[0106] Specifically, the job success rate, the duration of historical problem repairs, and the runtime corresponding to the log data records are simultaneously input into the log analysis model for calculation. The result of the job log data analysis is obtained by combining the table name or file name with the set of file size and file transcoding and transmission time.

[0107] For example, the job success rate (m / M) × 100%, the historical problem repair duration T2, and the runtime T1 corresponding to the log data record are simultaneously input into the log analysis model for calculation. Combined with the table name or file name, the set of file size and file transcoding and transmission time is obtained, which is the job log data analysis result.

[0108] In this embodiment, by describing the detailed process and input parameters of the operation log data analysis results, the influencing factors of the operation log data analysis results can be accurately reflected, which helps to set the judgment classification for the first judgment in the subsequent process.

[0109] In one embodiment, such as Figure 5 As shown, if the first judgment result is that the target operation is not interrupted, the operation log data to be analyzed is input into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted operation log data, including:

[0110] Step 502: Obtain the table name or file name corresponding to the job log data to be analyzed.

[0111] The table name or file name can be the name of the table or file that the job log data to be analyzed is stored in. By locating the table name or file name, detailed information about the job log data to be analyzed can be obtained, including the job log data itself and its attributes.

[0112] Specifically, the server responds to the terminal's instructions, obtains the table name or file name containing the job log data to be analyzed from the terminal, further locates the file corresponding to the job log data to be analyzed, and stores the data in the server's storage unit after obtaining the data. When the data is needed, the server will retrieve the corresponding data from the storage unit and send it to the server's central processing unit for calculation.

[0113] For example, the server responds to the terminal's instructions and obtains from the terminal the table name c1 or file name c2 corresponding to the job log data C to be analyzed. The server locates the file corresponding to the job log data C to be analyzed based on the table name c1 or file name c2, and retrieves the data that needs to be used into the central processing unit. The data that does not need to be processed at the moment is stored in the server's storage unit.

[0114] Step 504: Input the table name or file name corresponding to the work log data to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted work log data.

[0115] Specifically, the table name or file name corresponding to the job log data to be analyzed is input into the alarm tracking model. Based on the information, content, and parameters in the table name or file name, the job log data to be analyzed in the table or file is judged to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data. If the second judgment result is a normal job, the information of the normal job (including job start time, runtime, data size, dependencies, etc.) is output and recorded in the normal job information table in the database. If the second judgment result is an abnormal alarm, the recent normal operation data and the current abnormal operation data of the job are transmitted to the visualization module.

[0116] For example, the table name c1 or file name c2 corresponding to the job log data C to be analyzed is input into the alarm tracking model for a second analysis and judgment. Since the job data to be analyzed entering the alarm tracking model was judged as uninterrupted in the first judgment, the second judgment result P is either a normal job or an abnormal alarm. If the second judgment result P is a normal job, the information of the normal job (including job start time, runtime, data size, dependencies, etc.) is output and recorded in the normal job operation information table in the database. If the second judgment result P is an abnormal alarm, the recent normal operation data and the current abnormal operation data of the job are transmitted to the visualization module.

[0117] In this embodiment, by describing the specific path for obtaining the second judgment result, the input and output parameters of the alarm tracking model can be clearly defined, indicating that there is a clear judgment mechanism for the specific judgment of the target operation that is not interrupted but has abnormal alarms or normal operation, and the reason for the abnormal alarm can be further understood.

[0118] In one embodiment, such as Figure 6 As shown, input the table name or file name corresponding to the work log data to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted work log data, including:

[0119] Step 602: Obtain the data volume corresponding to the job log data to be analyzed based on the table name or file name of the job log data to be analyzed.

[0120] The data volume can be the amount of data contained in each share of the job log data to be analyzed, or it can be considered as the amount of information contained in each share of the job log data to be analyzed.

[0121] Specifically, the server locates the table name or file name corresponding to the job log data to be analyzed, extracts the information from the table or file, obtains the data from the table or file, and performs classification, statistics and other processing on the data to obtain the data volume corresponding to the job log data to be analyzed.

[0122] For example, the server locates the table name c1 or file name c2 corresponding to the job log data C to be analyzed based on the information. According to the acquisition rules, the data in table name c1 or file name c2 is extracted, and the data corresponding to table name c1 or file name c2 is obtained. This data is part or all of the job log data C to be analyzed. After classifying, statistically processing and other methods, the amount of data corresponding to the job log data C to be analyzed is obtained.

[0123] Step 604: Based on the start time, runtime, the amount of data corresponding to the job log data to be analyzed, and environmental information, input into the abnormal state calculation formula to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data.

[0124] Among them, environmental information can include server availability and the running status of previous jobs. This information generally represents the current computing environment and the quantities that can be computed.

[0125] Among them, the abnormal state calculation formula can be an analysis formula used in the abnormal analysis model to integrate and comprehensively calculate the various data in the log data of the job to be analyzed, and then obtain the status of the target job.

[0126] Specifically, the start time, runtime, and data volume from the job log data that was not interrupted, along with the current server's environmental information, are input into the abnormal state calculation formula for a second analysis. The target job is then judged based on the analysis results. The second judgment result corresponding to the analysis results of the uninterrupted job log data is obtained, tagged as either "normal" or "uninterrupted but with abnormal runtime information." Jobs tagged as "normal" are classified as normal jobs, while those tagged as "uninterrupted but with abnormal runtime information" are classified as abnormal jobs.

[0127] For example, the start time t1, runtime T1, and data volume of the job log data C to be analyzed without interruption, along with the current server's environmental information (e.g., server availability, previous job running status), are input into the abnormal state calculation formula for a second analysis. The abnormal state calculation formula processes t1, T1, data volume, and environmental information to obtain the corresponding analysis results. Based on the analysis results, a second judgment is made to obtain the second judgment result P corresponding to the analysis result of the uninterrupted job log data.

[0128] In this embodiment, by inputting multiple different parameters into the abnormal state calculation formula, the second judgment result is obtained, which clarifies the method of obtaining the second judgment result. For the target operation whose second judgment result is an abnormal alarm, the parameters corresponding to the second judgment result can be used to reverse access to find the cause of the abnormal alarm.

[0129] In one embodiment, such as Figure 7 As shown, the method also includes:

[0130] Step 702: Input the task log data to be analyzed based on the target task output corresponding to the abnormal alarm into the tracking algorithm model to obtain the third judgment result.

[0131] The tracking algorithm model can monitor recent abnormal jobs. If the tracking algorithm model finds that a job with the second judgment result of being abnormal does not show any abnormal conditions after monitoring, it is reclassified as a normal job. If the tracking algorithm model finds the same conclusion as the second judgment result after monitoring, it is still classified as an abnormal job.

[0132] The third judgment result can be a new judgment result obtained after tracking the target operation whose second judgment result is abnormal. This judgment result can change the conclusion of the second judgment result to prevent the second judgment result from being misjudged and causing the data to be sent to the wrong target location.

[0133] Specifically, the task log data output by the target task corresponding to the abnormal alarm is input into the tracking algorithm model. The tracking algorithm model tracks and follows up within a preset time period to obtain the tracking result corresponding to the preset time period, which is the third judgment result. The third judgment result can be a normal task or an abnormal alarm, similar to the second judgment result. For the tracking algorithm, firstly, a list of abnormal tasks is retrieved from the task abnormal operation information table, and the task execution information for the day is obtained from the list. Then, it is determined whether the task is still in an abnormal state. If so, the task-related data in the task abnormal operation information table is updated; otherwise, the number of normal executions is accumulated, and it is determined whether it is still less than N (N successful executions are required) to update and remove the abnormal monitoring status of the task.

[0134] The formula for calculating abnormal states is as follows:

[0135]

[0136] Where S i S′ represents the i-th feature information in the latest running information of job S. ji This represents the i-th feature information in the j-th recent normal operation information of job S. This represents the average of the i-th feature information from the n most recent normal runs of job S. δ is a manually set threshold; exceeding the threshold indicates an anomaly, while falling below it indicates normal operation. The tracking algorithm model logic diagram is as follows: Figure 11 As shown.

[0137] For example, for a target job with an abnormal alarm as the second judgment result P, the job log data to be analyzed corresponding to the target job is input into the tracking algorithm model for tracking and follow-up for a period of time. This period of time can be a preset time, and the corresponding third judgment result Q is obtained after tracking and follow-up.

[0138] Step 704: If the third judgment result is abnormal, then update the corresponding running job data in the abnormal running information table.

[0139] The abnormal operation information table can be used to record abnormal information corresponding to the target job, such as: low success rate, mismatched start time, excessive runtime, unavailable server status, incomplete previous job operation status, excessive repair time, etc.

[0140] Specifically, if the result of the third judgment after the third judgment is an abnormal alarm, the running job data in the abnormal operation information table generated by the second judgment result of an abnormal alarm will be updated and overwritten by the running job data generated by the third judgment result.

[0141] For example, if the third judgment result Q obtained after the third judgment is an abnormal alarm, and the corresponding running job data q is generated at the same time, then the running job data q is used to overwrite the running job data p generated when the second judgment result P is also an abnormal alarm.

[0142] Step 706: If the third judgment result is normal, repeat the step of inputting the target job log data to be analyzed into the tracking algorithm model corresponding to the abnormal alarm, and record the number of times the step is repeated.

[0143] Specifically, if the third judgment result is normal, the log data of the target job to be analyzed corresponding to the target job whose second judgment result is abnormal will be repeatedly input into the tracking algorithm model for multiple calculations. Here, multiple calculations are greater than or equal to 2, which is generally a preset threshold N, and the number of repeated runs will be recorded.

[0144] For example, if the third judgment result Q is a normal operation, then the log data C of the target operation to be analyzed corresponding to the second judgment result P is abnormal is re-input into the tracking algorithm model, and a threshold N is set for the number of times this step is repeated. After multiple runs, the number of repetitions N' is recorded.

[0145] In this embodiment, by inputting the work log data to be analyzed corresponding to the abnormal alarm into the tracking algorithm model for multiple analysis and calculation, a third judgment result is obtained. This enables multiple tracking and analysis of the work log data to be analyzed with abnormal alarms, avoiding errors in the second judgment result that could lead to deviations in the data flow destination.

[0146] In one embodiment, such as Figure 8 As shown, the target job log data to be analyzed is repeatedly input into the tracking algorithm model, corresponding to the repeated execution of the abnormal alarm, to obtain the fourth judgment result, including:

[0147] Step 802: If the number of times is less than the preset threshold, the output of the task log data to be analyzed for the target task is judged as abnormal again.

[0148] Specifically, if the number of times the task log data to be analyzed is repeatedly input into the tracking algorithm model is less than a preset threshold and a third judgment result of abnormal alarm occurs, then the tracking algorithm model will not be used for tracking and follow-up. Instead, the original judgment result of the task log data to be analyzed output for the target task will be maintained, and an abnormal alarm will be issued.

[0149] For example, if the job log data C to be analyzed is repeatedly input into the tracking algorithm model 8 times, the third judgment result Q will be abnormal again. The set threshold for the number of repetitions is 10. Since the number of repetitions is less than the preset 10, the original judgment result will still be maintained for the job log data C repeatedly input into the tracking algorithm model, which is an abnormal alarm.

[0150] Step 804: If the number of occurrences is greater than or equal to the number of occurrences threshold, then the abnormal monitoring status corresponding to the target job is lifted, and normal job information is output.

[0151] Specifically, if the number of times the task log data to be analyzed is repeatedly input into the tracking algorithm model exceeds the preset threshold, and the third judgment result is always normal, then through the tracking and follow-up of the tracking algorithm model, the task log data to be analyzed output for the target task will change the original judgment result, that is, the original abnormal alarm will be changed into normal operation.

[0152] For example, if the job log data C to be analyzed is repeatedly input into the tracking algorithm model 10 times, and the third judgment result Q is always output as normal operation, and the threshold for the number of repetitions is set to 10, since 10 repetitions equals the preset 10, the original judgment result for the repeatedly input job log data C into the tracking algorithm model will be changed, that is, the abnormal alarm will be changed to normal operation.

[0153] In this embodiment, by comparing the number of loop calculations of the analysis log data of abnormal alarms with a preset value, it can be determined that the log data corresponding to the threshold number of loop calculations is a normal operation, while the log data corresponding to the threshold number of loop calculations is an abnormal operation.

[0154] In one embodiment, this job log monitoring method further includes: a visualization component that can display various relevant information about job execution, such as job start time, runtime, data volume, and file size, according to user needs. Error analysis and proposed solutions for interrupted jobs are also provided below, which can be used as a reference for developers and can be helpful for new employees or developers who have just joined the project.

[0155] The flowchart of the intelligent log monitoring system based on attention mechanism and neural network, which outlines the logic of this job log monitoring scheme, is as follows: Figure 12 As shown, a log analysis module is first proposed, which uses a deep bidirectional recurrent neural network model with an attention mechanism as its main component. This module analyzes the logs of application jobs in real time, monitors the daily running status of jobs, and records them in a database. The bidirectional recurrent neural network performs deep contextual parsing of the logs, and the attention mechanism further enhances the network's ability to parse large volumes of log data, thereby improving analysis efficiency and achieving real-time or near-real-time analysis. The module categorizes and organizes the running information of each job, directly transmitting information from normal jobs to the database and writing it into the normal job running information table. The analysis results of abnormal jobs are transmitted to the next module for further processing.

[0156] A new alarm tracking module is proposed, which classifies abnormal jobs based on an alarm algorithm. Interrupted jobs trigger an alarm immediately, and error logs are captured, relevant data (success rate, response time, etc.) are collected and compared with recent normal operation data for that job. The comparison results are then transmitted to the visualization module. For jobs that are not interrupted but have abnormal operation information, an alert is issued, relevant data (success rate, response time, etc.) are collected, and compared with recent normal operation data for that job. The comparison results are then transmitted to the visualization module. Based on the tracking algorithm, monitored jobs are given priority monitoring and analysis. After N normal operation cycles, the monitoring status of the job is deactivated.

[0157] Finally, a visualization module is proposed to visualize job execution information and display it in chart form. The chart format can also be changed according to specific user needs.

[0158] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] Based on the same inventive concept, this application also provides a work log monitoring device for implementing the work log monitoring method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more work log monitoring device embodiments provided below can be found in the limitations of the work log monitoring method described above, and will not be repeated here.

[0160] In one embodiment, such as Figure 13 As shown, a job log monitoring device is provided, comprising: a job log data acquisition module, a first judgment result acquisition module, and an anomaly alarm module, wherein:

[0161] The task log data acquisition module 1302 is used to acquire the task log data to be analyzed from the task log center; the task log data to be analyzed is the data obtained after processing the log data generated by the target task during runtime.

[0162] The first judgment result acquisition module 1304 is used to input the job log data to be analyzed into the log analysis model, obtain the job log data analysis result, and judge whether the target job is interrupted based on the job log data analysis result to obtain the first judgment result; the log analysis model is a multi-layer structure bidirectional gated cyclic unit network;

[0163] The abnormal alarm module 1306 is used to input the operation log data to be analyzed into the alarm tracking model if the first judgment result is that the target operation is not interrupted, and obtain the second judgment result corresponding to the analysis result of the uninterrupted operation log data; if the second judgment result is an abnormal alarm, it outputs abnormal operation data for the target operation.

[0164] In one embodiment, the first judgment result obtaining module is used to obtain the log data record corresponding to the job name of the target job, the start time and end time of the log data record; input the log data record corresponding to the job name, the start time and end time of the log data record into the log analysis model to obtain the job log data analysis result; and judge whether the target job is interrupted based on the job log data analysis result to obtain the first judgment result.

[0165] In one embodiment, the first judgment result obtaining module is used to calculate the job success rate and historical problem repair duration for the time period corresponding to the job name based on the log data record corresponding to the job name; obtain the runtime corresponding to the log data record based on the start time and end time corresponding to the log data record; and input the job success rate, historical problem repair duration and runtime corresponding to the log data record into the log analysis model to obtain the job log data analysis result.

[0166] In one embodiment, the second judgment result obtaining module is used to obtain the table name or file name corresponding to the job log data to be analyzed, the table name or file name reflecting the carrying status of the job log data to be analyzed; input the table name or file name corresponding to the job log data to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the uninterrupted job log data analysis result; the second judgment result is used to characterize whether there is an abnormal alarm in the target job.

[0167] In one embodiment, the second judgment result obtaining module is used to obtain the data volume corresponding to the job log data to be analyzed based on the table name or file name of the job log data to be analyzed; and to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data based on the start time, runtime, data volume corresponding to the job log data to be analyzed and environmental information input into the abnormal state calculation formula.

[0168] In one embodiment, the tracking module is used to input the analysis log data of the target job output corresponding to the abnormal alarm into the tracking algorithm model to obtain a third judgment result; if the third judgment result is abnormal, the running job data corresponding to the abnormal operation information table is updated; the abnormal operation information table is used to record the abnormal information corresponding to the target job; if the third judgment result is normal, the step of inputting the analysis log data of the target job output corresponding to the abnormal alarm into the tracking algorithm model is repeated, and the number of times the step is repeated is recorded.

[0169] In one embodiment, the tracking module is configured to, if the number of occurrences is less than a preset threshold, output the task log data to be analyzed for the target task again and determine it as abnormal; if the number of occurrences is greater than or equal to the threshold, remove the abnormal monitoring status corresponding to the target task and output normal task information.

[0170] Each module in the aforementioned job log monitoring device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0171] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores server data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a job log monitoring method.

[0172] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0174] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0175] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for monitoring job logs, characterized in that, The method includes: Obtain the job log data to be analyzed from the job log center; the job log data to be analyzed is the data obtained after processing the log data generated by the target job during runtime, and the job log data to be analyzed includes job name, start time, end time, and table name or file name; The task log data to be analyzed is input into the log analysis model to obtain the task log data analysis result. Based on the task log data analysis result, it is determined whether the target task is interrupted to obtain a first judgment result. The log analysis model is a multi-layer bidirectional gated cyclic unit network. The step of inputting the job log data to be analyzed into the log analysis model to obtain the job log data analysis results includes: Based on the log data records corresponding to the job name, calculate the job success rate and historical problem repair duration for the time period corresponding to the job name; Based on the start time and end time corresponding to the log data record, the runtime corresponding to the log data record is obtained; The job success rate, the duration of historical problem repairs, and the runtime corresponding to the log data records are input into the log analysis model to obtain the job log data analysis results. If the first judgment result is that the target operation is not interrupted, the log data of the operation to be analyzed is input into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted operation log data; if the second judgment result is an abnormal alarm, abnormal operation data is output for the target operation. The step of inputting the work log data to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted work log data includes: Based on the table name or file name of the job log data to be analyzed, the data volume corresponding to the job log data to be analyzed is obtained; Based on the start time, the runtime, the amount of data corresponding to the job log data to be analyzed, and the environmental information input into the abnormal state calculation formula, a second judgment result corresponding to the analysis result of the uninterrupted job log data is obtained, wherein the environmental information is the server availability status and the running status of the previous job.

2. The method according to claim 1, characterized in that, The process involves inputting the task log data to be analyzed into the log analysis model to obtain task log data analysis results. A first judgment result is obtained based on whether the task log data analysis results are interrupted, including: Obtain the log data record corresponding to the job name of the target job, and the start time and end time of the log data record; The log data record corresponding to the job name, the start time and end time corresponding to the log data record are input into the log analysis model to obtain the job log data analysis result. Based on the analysis results of the job log data, a judgment is made as to whether the target job is interrupted, and the first judgment result is obtained.

3. The method according to claim 1, characterized in that, If the first judgment result indicates that the target operation is not interrupted, then the log data of the operation to be analyzed is input into the alarm tracking model to obtain a second judgment result corresponding to the analysis result of the uninterrupted operation log data, including: Obtain the table name or file name corresponding to the job log data to be analyzed, where the table name or file name reflects the carrying status of the job log data to be analyzed. The table name or file name corresponding to the job log data to be analyzed is input into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data; the second judgment result is used to characterize whether there is an abnormal alarm in the target job.

4. The method according to claim 1, characterized in that, The method further includes: The task log data to be analyzed, based on the output of the target task corresponding to the abnormal alarm, is input into the tracking algorithm model to obtain a third judgment result; If the third judgment result is abnormal, then the corresponding running job data in the abnormal running information table is updated; the abnormal running information table is used to record the abnormal information corresponding to the target job; If the third judgment result is normal, then repeat the step of inputting the log data of the job to be analyzed into the tracking algorithm model corresponding to the abnormal alarm, and record the number of times the step is repeated.

5. The method according to claim 4, characterized in that, The process involves repeatedly executing the target job output corresponding to the abnormal alarm, inputting the job log data to be analyzed into the tracking algorithm model, and obtaining a fourth judgment result, including: If the number of times is less than the preset threshold, the log data of the job to be analyzed will be output again for the target job and judged as abnormal. If the number of occurrences is greater than or equal to the threshold number, the abnormal monitoring status corresponding to the target operation is lifted, and normal operation information is output.

6. A work log monitoring device, characterized in that, The device includes: The task log data acquisition module is used to acquire task log data to be analyzed from the task log center. The task log data to be analyzed is the data obtained after processing the log data generated by the target task during runtime. The task log data to be analyzed includes the task name, start time, end time, and table name or file name. The first judgment result acquisition module is used to input the job log data to be analyzed into the log analysis model to obtain the job log data analysis result, and to judge whether the target job is interrupted based on the job log data analysis result to obtain the first judgment result; the log analysis model is a multi-layer bidirectional gated cyclic unit network; wherein, inputting the job log data to be analyzed into the log analysis model to obtain the job log data analysis result includes: calculating the job success rate and historical problem repair duration for the time period corresponding to the job name based on the log data record corresponding to the job name; obtaining the runtime corresponding to the log data record based on the start time and end time corresponding to the log data record; inputting the job success rate, the historical problem repair duration, and the runtime corresponding to the log data record into the log analysis model to obtain the job log data analysis result; An anomaly alarm module is used to input the log data of the job to be analyzed into an alarm tracking model if the first judgment result is that the target job is not interrupted, and obtain a second judgment result corresponding to the analysis result of the uninterrupted job log data; if the second judgment result is an anomaly alarm, output abnormal operation data for the target job; wherein, inputting the log data of the job to be analyzed into the alarm tracking model to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data includes: obtaining the data volume corresponding to the log data of the job to be analyzed based on the table name or file name of the log data of the job to be analyzed; inputting the start time, the runtime, the data volume corresponding to the log data of the job to be analyzed, and environmental information into an anomaly state calculation formula to obtain the second judgment result corresponding to the analysis result of the uninterrupted job log data, wherein the environmental information is the server availability status and the running status of the previous job.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images