Abnormal operation identification method and device, computer device and storage medium
By extracting critical paths from the execution logic path of batch jobs and generating a list of jobs and business data to be detected, abnormal jobs can be automatically identified, solving the problems of high manpower and material costs and lack of objectivity in existing technologies, and improving identification efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2022-05-16
- Publication Date
- 2026-06-02
AI Technical Summary
Existing batch job testing methods are labor-intensive and lack objectivity, and cannot effectively identify business logic conflicts between batch jobs.
By extracting the critical path in the execution logic path of batch jobs, the set of target jobs is obtained, and a list of jobs to be checked is generated to correspond to business data. Anomaly detection is then performed to identify abnormal jobs.
It enables automated identification of abnormal jobs in batch operations, improving identification efficiency, reducing manpower and material resources, and enhancing the objectivity of identification.
Smart Images
Figure CN114860404B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to an abnormal operation identification method, apparatus, computer equipment, storage medium and computer program product. Background Technology
[0002] With the rapid development of various business operations within an enterprise, the amount of data in business systems is increasing daily. When data migration is required between business systems, batch jobs are used for data scheduling to achieve large-scale data export, processing, and import. However, due to the massive amount of data, conflicts may exist in the business execution logic (e.g., data scheduling) between pre-developed batch jobs, thus affecting the overall data migration efficiency. Therefore, during the development phase of batch jobs, thorough testing is necessary.
[0003] Current batch job testing methods involve technical personnel using their own testing experience to analyze the scheduling relationships between each job, manually testing and identifying potential business logic conflicts in program calls and data calls between batch jobs.
[0004] However, current batch testing methods are labor-intensive, resource-intensive, and lack objectivity. Summary of the Invention
[0005] Therefore, it is necessary to provide an abnormal operation identification method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems.
[0006] Firstly, this application provides a method for identifying abnormal operations. The method includes:
[0007] Extract the critical path from the execution logic path formed by batch jobs; the critical path contains multiple target jobs.
[0008] Obtain the job set corresponding to each of the target jobs;
[0009] Based on the job set, generate a list of tasks to be tested that includes the correspondence between the tasks in the job set and the business data.
[0010] Anomaly detection is performed based on the list of tasks to be detected, and the abnormal task identification results of the batch tasks are determined.
[0011] In one embodiment, extracting the critical path from the execution logic path constituted by the batch jobs includes:
[0012] In the execution logic path formed by batch jobs, determine the execution logic relationship between each job and other jobs in the batch job;
[0013] Based on the preset critical path algorithm and the determined execution logic relationship between each job, the critical path containing the target job is obtained.
[0014] In one embodiment, the job set includes a target job set and a non-target job set, and obtaining the job set corresponding to each target job includes:
[0015] In the execution logic path formed by the batch jobs, the preceding job, the following job, and the non-related job corresponding to each target job are determined, and corresponding sets of preceding jobs, following jobs, and non-related jobs are generated respectively; the jobs included in the non-critical job set are jobs that do not have preceding or following execution logic with the target job.
[0016] The target set of tasks is obtained by taking the union of the previous set of tasks and the subsequent set of tasks, and the set of non-associated tasks is determined as the non-target set of tasks.
[0017] In one embodiment, the job set includes a target job set and a non-target job set. The step of generating a list to be detected, containing the correspondence between jobs and business data in the job set, includes:
[0018] Based on the target job set, determine the first business data corresponding to each associated job contained in the target job set, and generate a target data to be detected list containing each associated job and the first business data;
[0019] Based on the set of non-target jobs, determine the second business data corresponding to each non-related job contained in the set of non-target jobs, and generate a list of non-target data to be detected that includes each non-related job and the second business data.
[0020] In one embodiment, the step of performing anomaly detection based on the list to be detected and determining the anomaly identification result of the batch job includes:
[0021] The intersection of the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list is performed to obtain the first data set to be detected;
[0022] The business data contained in the first set of data to be detected is identified as abnormal business data, and the job corresponding to the abnormal business data is identified as abnormal job, generating an abnormal job identification result.
[0023] In one embodiment, the target job set includes related jobs divided into preceding jobs and subsequent jobs, and the target data to be detected list includes a preceding target data to be detected list and a subsequent target data to be detected list. The step of performing anomaly detection based on the to-be-detected list to determine the abnormal job identification result of the batch jobs includes:
[0024] The intersection of the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list is performed to obtain the first data set to be detected;
[0025] The intersection of the preceding related business data in the preceding target data to be tested list and the following related business data in the following target data to be tested list is processed to obtain the second set of data to be tested.
[0026] The business data contained in the first and second sets of data to be detected are identified as abnormal business data, and the jobs corresponding to the abnormal business data are identified as abnormal jobs, generating abnormal job identification results.
[0027] Secondly, this application also provides an abnormal operation identification device. The device includes:
[0028] The extraction module is used to extract the critical path from the execution logic path composed of batch jobs; the critical path contains multiple target jobs.
[0029] The acquisition module is used to acquire the job set corresponding to each of the target jobs;
[0030] The generation module is used to generate a list of tasks to be tested, which contains the correspondence between tasks and business data in the task set, based on the task set.
[0031] The determination module is used to perform anomaly detection based on the list to be detected and determine the abnormal job identification result of the batch job.
[0032] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0033] Extract the critical path from the execution logic path formed by batch jobs; the critical path contains multiple target jobs.
[0034] Obtain the job set corresponding to each of the target jobs;
[0035] Based on the job set, generate a list of tasks to be tested that includes the correspondence between the tasks in the job set and the business data.
[0036] Anomaly detection is performed based on the list of tasks to be detected, and the abnormal task identification results of the batch tasks are determined.
[0037] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0038] Extract the critical path from the execution logic path formed by batch jobs; the critical path contains multiple target jobs.
[0039] Obtain the job set corresponding to each of the target jobs;
[0040] Based on the job set, generate a list of tasks to be tested that includes the correspondence between the tasks in the job set and the business data.
[0041] Anomaly detection is performed based on the list of tasks to be detected, and the abnormal task identification results of the batch tasks are determined.
[0042] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0043] Extract the critical path from the execution logic path formed by batch jobs; the critical path contains multiple target jobs.
[0044] Obtain the job set corresponding to each of the target jobs;
[0045] Based on the job set, generate a list of tasks to be tested that includes the correspondence between the tasks in the job set and the business data.
[0046] Anomaly detection is performed based on the list of tasks to be detected, and the abnormal task identification results of the batch tasks are determined.
[0047] The aforementioned abnormal job identification method, apparatus, computer equipment, storage medium, and computer program product extract a critical path from the execution logic path composed of batch jobs. The critical path contains multiple target jobs. A job set corresponding to each target job is obtained. Based on the job set, a checklist containing the correspondence between jobs and business data in the job set is generated. Anomaly detection is performed based on the checklist to determine the abnormal job identification result of the batch jobs. This method extracts a critical path from the execution logic path composed of batch jobs, obtains a job set corresponding to each target job in the critical path, and generates a checklist containing each job and its corresponding business data based on the job set. Then, based on this checklist, execution logic anomalies during the batch job execution process are detected, achieving automated determination of abnormal job identification results and improving the efficiency of abnormal job identification. Attached Figure Description
[0048] Figure 1 This is a flowchart illustrating an abnormal job identification method in one embodiment;
[0049] Figure 2 This is a flowchart illustrating the steps for extracting the critical path in one embodiment;
[0050] Figure 3 This is a schematic diagram illustrating the directed relationship of batch job execution logic in one embodiment;
[0051] Figure 4 This is a flowchart of the steps for generating a target job set and a non-target job set in one embodiment;
[0052] Figure 5 This is a flowchart illustrating the steps of generating a list of data to be tested in one embodiment;
[0053] Figure 6 This is a flowchart illustrating the steps for generating abnormal job identification results in one embodiment;
[0054] Figure 7 This is a flowchart illustrating the steps for generating abnormal job identification results in another embodiment;
[0055] Figure 8 This is a structural block diagram of an abnormal operation identification device in one embodiment;
[0056] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0058] In one embodiment, such as Figure 1 As shown, an abnormal operation identification method is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the executing entity of this solution is not limited and is collectively referred to as a computer device. The method includes the following steps:
[0059] Step 102: Extract the critical path from the execution logic path formed by the batch jobs.
[0060] The critical path contains multiple target tasks.
[0061] In practice, the business systems deployed on computer equipment process a large amount of business data daily. When there are upgrades to old and new systems that require the migration of large amounts of data from these systems, the computer equipment needs to use batch jobs (or simply jobs) to achieve this. In batch jobs, the computer equipment controls the invocation of programs for each job based on preset business execution logic. These programs then reference (or call) relevant business data to complete the data migration. Therefore, there are business execution logic relationships between batch jobs, forming execution paths (also called execution logic paths) in the computer equipment that are related by business execution logic. When it is necessary to identify abnormal jobs in batch jobs, the computer equipment extracts the critical path from the execution logic path formed by the batch jobs, based on the longest processing latency. Each job contained in the critical path is the target job that needs to be analyzed and detected independently.
[0062] Step 104: Obtain the set of jobs corresponding to each target job.
[0063] In implementation, after extracting the critical path from each execution logic path corresponding to all batch jobs, the computer device uses each target object contained in the critical path as the anomaly detection benchmark job to determine the job set corresponding to the anomaly detection benchmark job (i.e. the target job) in all batch jobs. That is, all jobs contained in the job set are based on the target job to determine the execution logic relationship of other jobs in the batch jobs relative to the target job, such as whether they are related to or not related to the target job.
[0064] Step 106: Generate a list of tasks to be tested, which contains the correspondence between tasks and business data in the task set, based on the task set.
[0065] In implementation, the computer equipment analyzes the set of jobs corresponding to each target job. Based on the calling relationship between jobs and programs (also known as the correspondence relationship) and the calling relationship between programs and business data (also known as the correspondence relationship), the computer equipment determines the correspondence relationship between each job in the job set and the business data. Then, the computer equipment generates a list of jobs to be tested that contains the correspondence relationship between jobs and business data.
[0066] Step 108: Perform anomaly detection based on the list of tasks to be detected, and determine the anomaly identification results for the batch operations.
[0067] In implementation, since the list to be tested contains the correspondence between each job and business data, the computer device can identify the business execution logic and data call logic of each target job's list of jobs to be tested by performing anomaly identification (i.e., call conflict identification). The computer device can determine whether there are conflicts in the business execution logic and data call logic of jobs that are related to or not related to the target job, based on the target job as the business execution benchmark job. Then, after completing the anomaly detection analysis of the list of jobs to be tested for all target jobs in the critical path, the computer device can determine the anomaly job identification results corresponding to the overall batch jobs, that is, whether there are anomaly jobs with business execution logic conflicts in the batch jobs.
[0068] In the aforementioned abnormal job identification method, a critical path is extracted from the execution logic path composed of batch jobs. This critical path contains multiple target jobs. The computer device acquires the job set corresponding to each target job. Then, based on the job set, the computer device generates a checklist containing the correspondence between jobs and business data within the job set. Finally, the computer device performs anomaly detection based on the checklist to determine the abnormal job identification result for the batch jobs. This method extracts the critical path from the execution logic path composed of batch jobs, obtains the job set corresponding to each target job within the critical path, and generates a checklist containing each job and its corresponding business data based on the job set. Then, based on this checklist, it detects whether the execution logic during the batch job execution process is abnormal, thus achieving automated determination of abnormal job identification results and improving the efficiency of abnormal job identification.
[0069] In one embodiment, such as Figure 2 As shown, the specific processing procedure of step 102 includes the following steps:
[0070] Step 202: In the execution logic path formed by batch jobs, determine the execution logic relationship between each job and other jobs in the batch job.
[0071] In implementation, within the execution logic path comprised of batch jobs, the computer device analyzes and determines the execution logic relationship between each job and all other jobs in the batch. Specifically, the possible execution logic relationships between a job and other jobs can be categorized into several types, such as associated relationships and non-associated relationships. Associated relationships can be further divided into preceding and subsequent associated relationships based on the order of execution logic; therefore, two jobs being in the same execution logic path is a prerequisite for having an associated relationship. A non-associated relationship occurs when the logical execution relationships between two jobs are not in the same execution logic path. This application embodiment does not limit the execution logic relationships that exist between jobs. In the process of analyzing the execution logic relationships between batch jobs, the computer device can also determine whether there are any conflicts in execution logic relationships between jobs, and promptly detect abnormal jobs.
[0072] Step 204: Based on the preset critical path algorithm and the determined execution logic relationship between each job, obtain the critical path containing the target job.
[0073] In practice, the computer equipment calculates the path with the longest latency among the execution logic paths based on the preset critical path algorithm and the determined execution logic relationship between each job. This path is used as the critical path for batch job execution, and the jobs contained in this critical path are the target jobs.
[0074] Specifically, such as Figure 3 As shown, Figure 3 Each node in the directed graph represents a batch job, and among the batch jobs required for a single data migration, there is at least one batch opening job and one batch closing job. Figure 3 The entire set of jobs includes {JOBA, JOBB, JOBC, JOBD, JOBE, JOBF, JOBG, JOBZ}, where JOBA is a batch opening job and JOBZ is a batch closing job. These jobs have different execution logic relationships, forming three execution logic paths. The computer uses a pre-defined critical path algorithm to determine the critical path from these three paths. The critical path is determined by the path with the longest latency. Therefore, even if there are execution logic paths with the same number of batch jobs, the critical path can be determined because each job has a different execution time. For example, in these jobs, JOBB's execution time is longer than JOBC's, and JOBC's execution time is longer than JOBF's. Therefore, in... Figure 3 The critical path can be identified from the three given execution logic paths as JOBA-JOBB-JOBD-JOBE-JOBZ.
[0075] In one embodiment, such as Figure 4 As shown, the set of jobs corresponding to the target job includes the target job set and the non-target job set. The specific processing steps of step 104 include:
[0076] Step 402: In the execution logic path composed of batch jobs, determine the preceding job, following job and non-related job corresponding to each target job, and generate the corresponding preceding job set, following job set and non-related job set respectively.
[0077] The non-critical task set includes tasks that do not have any preceding or succeeding execution logic with the target task.
[0078] In implementation, within the execution logic path comprised of batch jobs, the computer equipment, based on the execution logic relationships between each batch job and using each target job as a reference, determines the preceding jobs, succeeding jobs, and non-related jobs corresponding to that target job. Then, for all preceding jobs corresponding to the target job, the computer equipment generates a set of preceding jobs (SET). i1 For all subsequent tasks corresponding to the target task, generate a set of subsequent tasks (SET). i2 Similarly, for all non-associated jobs of the target job, generate a set of non-associated jobs SET. i3 .
[0079] like Figure 3As shown, after the computer equipment determines the critical path, each batch of jobs within the critical path is considered a target job. Taking the target job JOBD in the critical path as an example, the computer equipment determines that the preceding jobs of the target job JOBD are JOBA, JOBB, and JOBC. Then, it generates a set of preceding jobs {JOBA, JOBB, JOBC} corresponding to JOBD. Here, preceding jobs refer to jobs whose execution order in the execution logic path of JOBD is before that of the target job JOBD. During this process, the computer equipment can identify whether there are any business execution logic conflicts for the target job JOBD. Similarly, for the target job JOBD, its corresponding subsequent jobs are JOBE and JOBZ. Here, subsequent jobs refer to jobs whose execution order in the execution logic relationship is after that of the target job. It is worth noting that the aforementioned preceding or succeeding tasks for the target task are not limited to those on the critical path. Any task that has a relationship with the target task (i.e., is on a certain execution logic path) can be identified as a preceding or succeeding task based on its execution order. For example, the preceding tasks of the JOBD target task on the critical path only include JOBA and JOBB, but on another execution logic path, JOBC also has a relationship with JOBD (also called an execution logic relationship). Therefore, the preceding task set of JOBD includes {JOBA, JOBB, JOBC}. Similarly, the method for determining the succeeding task set corresponding to the target task is the same as the method for determining the preceding tasks, and will not be repeated in the embodiments of this application.
[0080] Non-associative jobs for the target job refer to non-associative jobs that are not in any execution logical path with the target job in each batch job, that is, there is no preceding or succeeding execution logical relationship between them. Taking the JOBD target job as an example, the non-associative jobs of JOBD are JOBF and JOBZ. Therefore, the set of non-associative jobs generated by the computer device is {JOBF, JOBZ}.
[0081] Step 404: Take the union of the previous set of tasks and the subsequent set of tasks to obtain the target set of tasks, and determine the set of non-related tasks as the non-target set of tasks.
[0082] In implementation, the computer device takes the union of the sets of preceding and succeeding tasks corresponding to each target task to obtain the target task set. This target task set contains all associated tasks corresponding to the target task. Furthermore, the computer device determines the set of non-associated tasks as the non-target task set. Thus, when performing conflict analysis (i.e., abnormal task identification) of batch tasks, the computer device can sort and judge based on the target task set and the non-target task set determined with the target task as the reference benchmark.
[0083] In one embodiment, such as Figure 5 As shown, for each target object, the computer device obtains a set of jobs for that target object. This set of jobs includes a target job set and a non-target job set. The specific processing steps of step 106 include:
[0084] Step 502: Based on the target job set, determine the first business data corresponding to each associated job contained in the target job set, and generate a target data to be tested list containing each associated job and the first business data.
[0085] During implementation, in addition to execution logic conflicts between batch jobs, conflicts in business data calls may also occur during data migration. These can cause anomalies in the processing of batch jobs. Therefore, in addition to analyzing the job set corresponding to the target job, it is necessary to further analyze the corresponding business data. When data migration is performed, the computer equipment controls each batch job to call relevant functional programs. Each called program calls the corresponding business data based on its own execution logic. Thus, a correspondence exists between batch jobs and business data. For the target job set in the job set, the computer equipment determines the first business data corresponding to each batch job (also called associated job), and then generates a target data to be tested list JB2TBL containing the correspondence between each associated job and the first business data. i1 .
[0086] Step 504: Based on the non-target job set, determine the second business data corresponding to each non-related job contained in the non-target job set, and generate a non-target data to be detected list containing each non-related job and the second business data.
[0087] In implementation, based on the same processing procedure as the target job set mentioned above, the computer device determines the second business data corresponding to each non-associated job in the non-target job set. Then, the computer device generates a non-target data to-be-detected list JB2TBL containing the correspondence between each non-associated job and the second business data. i2 .
[0088] In one embodiment, such as Figure 6As shown, the list to be tested contains the correspondence between jobs and business data. Therefore, for the determined list to be tested, the abnormal conflicts of business execution logic between jobs and data calling logic can be analyzed. The specific processing procedure of step 108 includes the following steps:
[0089] Step 602: Perform intersection processing on the first business data contained in the target data to be tested list and the second business data contained in the non-target data to be tested list to obtain the first data set to be tested.
[0090] During implementation, the computer equipment checks the target data to be tested list JB2TBL. i1 The first business data and non-target data to be tested included in the JB2TBL list i2 The intersection of the second business data contained therein is processed to obtain the first data set to be detected, SET. i_table The first set of data to be detected can be either an empty set or a non-empty set.
[0091] Step 604: Identify the business data contained in the first set of data to be detected as abnormal business data, identify the jobs corresponding to the abnormal business data as abnormal jobs, and generate abnormal job identification results.
[0092] In implementation, for the first set of data to be detected SET i_table The computer equipment performs analysis and processing. Since the target data to be detected list represents the set of related jobs and the first set of business data corresponding to the target job, while the non-target data to be detected list represents the set of non-related jobs and the second set of business data corresponding to the target job, there is no logical constraint between related and non-related jobs relative to the target job. Therefore, there may be logical conflicts between the two, and consequently, there may be data processing logic conflicts for business data shared by related and non-related jobs. Therefore, if the first set of data to be detected is empty, it indicates that there is no data that both related and non-related jobs need to call. There are no conflicts between related and non-related jobs in terms of job execution logic and data call processing, meaning there are no abnormal jobs.
[0093] If the first set of data to be detected is not empty, it means that there is business data that both related and unrelated jobs need to access. In this case, there may be data access conflicts in the first set of data to be detected. Therefore, the computer device will set the first set of data to be detected as SET. i_tableThe business data contained therein is identified as abnormal business data. Further, the computer device can further reverse the process to identify related and unrelated jobs that both need to call the abnormal business data. Then, the computer device determines whether there is a data call processing conflict based on the execution time of the related and unrelated jobs. If the corresponding related and unrelated jobs call the business data (i.e., abnormal business data) at the same time, there is a data call processing conflict. The computer device identifies the related and unrelated jobs that are currently calling the abnormal business data as abnormal jobs and generates an abnormal job identification result containing the abnormal jobs.
[0094] If the corresponding associated and unassociated jobs do not call the business data at the same time, it means that there is no conflict in the execution logic of the associated and unassociated jobs or in the data call processing during the data migration process. The computer device determines that the current associated and unassociated jobs are non-abnormal jobs and generates an abnormal job identification result that does not contain abnormal jobs.
[0095] In another embodiment, such as Figure 7 As shown, in the execution logic path composed of batch jobs, not only may there be conflicts between related and unrelated jobs of the target job, but there is also a closed-loop conflict among the related jobs of the target job. That is, the preceding job, the target job, and the following job of the target job form a closed loop, causing the data migration task to get stuck in a closed loop and unable to complete smoothly. Therefore, in response to this situation, when identifying abnormal jobs using the list to be detected, the target data to be detected lists are generated separately for the preceding and following jobs in the related jobs for conflict analysis. Specifically, each related job in the target job set is divided into preceding jobs and following jobs, so the generated target data to be detected lists include the preceding target data to be detected list and the following target data to be detected list. The preceding target data to be detected list contains the correspondence between the preceding job and the preceding related business data, and the following target data to be detected list contains the correspondence between the following job and the following related business data. The specific processing of step 108 includes the following steps:
[0096] Step 702: Perform intersection processing on the first business data contained in the target data to be tested list and the second business data contained in the non-target data to be tested list to obtain the first data set to be tested.
[0097] During implementation, the computer equipment checks the target data to be tested list JB2TBL. i1 The first business data and non-target inspection list included in JB2TBL i2 The second set of business data contained therein is subjected to intersection processing to obtain the first set of data to be detected (SET). i_tableThe purpose of obtaining the first set of data to be detected in step 702 is the same as in step 602 above, which is to detect potential conflicts between related and unrelated jobs of the target job. This application embodiment will not elaborate further.
[0098] Step 704: Take the intersection of the preceding related business data in the preceding target data to be tested list and the subsequent related business data in the subsequent target data to be tested list to obtain the second set of data to be tested.
[0099] In practice, the computer equipment compares and analyzes the previous target data to be tested list and the subsequent target data to be tested list in the target data to be tested list. The intersection of the previous related business data in the previous target data to be tested list and the subsequent related business data in the subsequent target data to be tested list is processed to obtain the second set of data to be tested.
[0100] Step 706: Determine that the business data contained in the first and second sets of data to be detected are abnormal business data, identify the jobs corresponding to the abnormal business data as abnormal jobs, and generate abnormal job identification results.
[0101] In implementation, if the first set of data to be detected and / or the second set of data to be detected are non-empty sets, the computer device determines that the business data contained in the first set of data to be detected and the second set of data to be detected are abnormal business data, and identifies the job that calls the abnormal business data as an abnormal job, and generates an abnormal job identification result that contains the abnormal job in the execution logic relationship path.
[0102] Specifically, the first set of data to be detected, SET i_tableThe meaning of the data contained in the first data set is the same as in step 604 above. That is, if the first data set to be detected is a non-empty set, it means that the business data contained in the first data set to be detected may cause data call conflicts between related and unrelated jobs of the target job. Therefore, the data in the first data set to be detected is identified as abnormal business data. This embodiment of the application will not elaborate further. For the second data set to be detected, if the second data set to be detected is a non-empty set, the data in the second data set to be detected is that the preceding and subsequent jobs corresponding to the target job call the same business data. Such business data may cause a closed loop problem in the execution logic relationship between the target job, the preceding job, and the subsequent job. Therefore, the data in the second data set to be detected is identified as abnormal business data. The computer device can merge the abnormal business data contained in the first data set to be detected and the abnormal business data contained in the second data set to be detected to obtain a data set to be detected containing abnormal business data. Then, the computer device can reverse the mapping of the data set to be detected to the jobs that call abnormal business data, identify them as abnormal jobs, and generate an abnormal job identification result containing abnormal jobs.
[0103] If both the first and second sets of data to be detected are empty sets, then there is no abnormal business data. Consequently, the computer device generates an abnormal job identification result that does not contain abnormal jobs in the execution logic relationship path.
[0104] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0105] Based on the same inventive concept, this application also provides an abnormal operation identification device for implementing the abnormal operation identification method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the abnormal operation identification device provided below can be found in the limitations of the abnormal operation identification method described above, and will not be repeated here.
[0106] In one embodiment, such as Figure 8 As shown, an abnormal operation identification device 800 is provided, including: an extraction module 810, an acquisition module 820, a generation module 830, and a determination module 840, wherein:
[0107] Extraction module 810 is used to extract the critical path from the execution logic path composed of batch jobs; the critical path contains multiple target jobs.
[0108] The acquisition module 820 is used to acquire the set of jobs corresponding to each target job;
[0109] The generation module 830 is used to generate a list of tasks to be tested, which contains the correspondence between tasks and business data in the task set, based on the task set.
[0110] The determination module 840 is used to perform anomaly detection based on the list to be detected and to determine the anomaly identification results of the batch jobs.
[0111] In one embodiment, the extraction module 810 is specifically used to determine the execution logic relationship between each job and other jobs in the batch job in the execution logic path composed of batch jobs;
[0112] Based on the preset critical path algorithm and the determined execution logic relationship between each job, the critical path containing the target job is obtained.
[0113] In one embodiment, the job set includes a target job set and a non-target job set. The acquisition module 820 is specifically used to determine the preceding job, the following job, and the non-related job corresponding to each target job in the execution logic path composed of batch jobs, and generate the corresponding preceding job set, following job set, and non-related job set respectively. The jobs included in the non-critical job set are jobs that do not have preceding or following execution logic with the target job.
[0114] The target set of tasks is obtained by taking the union of the previous set of tasks and the subsequent set of tasks, and the set of non-related tasks is determined as the non-target set of tasks.
[0115] In one embodiment, the generation module 830 is specifically used to determine the first business data corresponding to each associated job contained in the target job set based on the target job set, and generate a target data to be detected list containing each associated job and the first business data.
[0116] Based on the set of non-target jobs, determine the second business data corresponding to each non-related job in the set of non-target jobs, and generate a list of non-target data to be tested that includes each non-related job and the second business data.
[0117] In one embodiment, the generation module 830 is specifically used to perform intersection processing on the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list to obtain the first data set to be detected.
[0118] The business data contained in the first set of data to be detected is identified as abnormal business data, and the jobs corresponding to the abnormal business data are identified as abnormal jobs, generating abnormal job identification results.
[0119] In one embodiment, each associated job in the target job set is divided into a preceding job and a subsequent job, and the target data to be detected list includes a preceding target data to be detected list and a subsequent target data to be detected list. The determining module 840 is specifically used to perform intersection processing on the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list to obtain the first data set to be detected.
[0120] The intersection of the preceding related business data in the preceding target data to be tested list and the subsequent related business data in the subsequent target data to be tested list is processed to obtain the second set of data to be tested.
[0121] The business data contained in the first and second sets of data to be detected are identified as abnormal business data, and the jobs corresponding to the abnormal business data are identified as abnormal jobs, generating abnormal job identification results.
[0122] Each module in the aforementioned abnormal operation identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0123] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an abnormal operation identification method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.
[0124] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0126] Extract the critical path from the execution logic path formed by batch jobs; the critical path contains multiple target jobs.
[0127] Obtain the set of jobs corresponding to each target job;
[0128] Based on the job set, generate a list of tasks to be tested that contains the correspondence between the jobs in the job set and the business data.
[0129] Anomaly detection is performed based on the list of tasks to be inspected to determine the anomaly identification results for batch operations.
[0130] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0131] In the execution logic path formed by batch jobs, determine the execution logic relationship between each job and other jobs in the batch job;
[0132] Based on the preset critical path algorithm and the determined execution logic relationship between each job, the critical path containing the target job is obtained.
[0133] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0134] In the execution logic path composed of batch jobs, the preceding job, following job, and non-related job corresponding to each target job are determined, and the corresponding sets of preceding jobs, following jobs, and non-related jobs are generated respectively; the jobs included in the non-critical job set are jobs that have no preceding or following execution logic with the target job.
[0135] The target set of tasks is obtained by taking the union of the previous set of tasks and the subsequent set of tasks, and the set of non-related tasks is determined as the non-target set of tasks.
[0136] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0137] Based on the target job set, determine the first business data corresponding to each associated job contained in the target job set, and generate a target data to be tested list containing each associated job and the first business data;
[0138] Based on the set of non-target jobs, determine the second business data corresponding to each non-related job in the set of non-target jobs, and generate a list of non-target data to be tested that includes each non-related job and the second business data.
[0139] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0140] The intersection of the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list is performed to obtain the first data set to be detected.
[0141] The business data contained in the first set of data to be detected is identified as abnormal business data, and the jobs corresponding to the abnormal business data are identified as abnormal jobs, generating abnormal job identification results.
[0142] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0143] The intersection of the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list is performed to obtain the first data set to be detected.
[0144] The intersection of the preceding related business data in the preceding target data to be tested list and the subsequent related business data in the subsequent target data to be tested list is processed to obtain the second set of data to be tested.
[0145] The business data contained in the first and second sets of data to be detected are identified as abnormal business data, and the jobs corresponding to the abnormal business data are identified as abnormal jobs, generating abnormal job identification results.
[0146] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. An abnormal work identification method characterized by comprising: The method includes: Extract the critical path from the execution logic path formed by batch jobs; the critical path contains multiple target jobs. Obtain the job set corresponding to each of the target jobs; Based on the job set, generate a list of tasks to be tested that includes the correspondence between the tasks in the job set and the business data. Anomaly detection is performed based on the list of tasks to be detected to determine the anomaly identification results of the batch operations; The job set includes a target job set and a non-target job set. The step of generating a checklist to be detected, containing the correspondence between jobs and business data in the job set, includes: Based on the target job set, determine the first business data corresponding to each associated job included in the target job set, and generate a target data to be detected list containing each associated job and the first business data; based on the non-target job set, determine the second business data corresponding to each non-associated job included in the non-target job set, and generate a non-target data to be detected list containing each non-associated job and the second business data. The step of performing anomaly detection based on the list of tasks to be detected, and determining the anomaly identification result of the batch jobs, includes: The intersection of the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list is performed to obtain a first data set to be detected; the business data contained in the first data set to be detected is identified as abnormal business data, and the job corresponding to the abnormal business data is identified as abnormal job, generating an abnormal job identification result.
2. The method of claim 1, wherein, Extracting the critical path from the execution logic path formed by batch jobs includes: In the execution logic path formed by batch jobs, determine the execution logic relationship between each job and other jobs in the batch job; Based on the preset critical path algorithm and the determined execution logic relationship between each job, the critical path containing the target job is obtained.
3. The method of claim 1, wherein, The step of obtaining the job set corresponding to each target job includes: In the execution logic path formed by the batch jobs, the preceding job, the following job, and the non-associated job corresponding to each target job are determined, and corresponding sets of preceding jobs, following jobs, and non-associated jobs are generated respectively; the jobs included in the non-associated job set are jobs that have no preceding or following execution logic with the target job. The target set of tasks is obtained by taking the union of the previous set of tasks and the subsequent set of tasks, and the set of non-associated tasks is determined as the non-target set of tasks.
4. The method of claim 1, wherein, The target job set includes related jobs divided into preceding jobs and subsequent jobs. The target data to be detected list includes a preceding target data to be detected list and a subsequent target data to be detected list. The step of performing anomaly detection based on the to-be-detected list to determine the abnormal job identification result of the batch jobs further includes: The intersection of the preceding related business data in the preceding target data to be tested list and the following related business data in the following target data to be tested list is processed to obtain the second set of data to be tested. The business data contained in the second set of data to be detected is determined to be abnormal business data, and the job corresponding to the abnormal business data is determined to be an abnormal job, and an abnormal job identification result is generated.
5. An abnormal work identification device characterized by comprising: The device includes: The extraction module is used to extract the critical path from the execution logic path composed of batch jobs; the critical path contains multiple target jobs. The acquisition module is used to acquire the job set corresponding to each of the target jobs; The generation module is used to generate a list of tasks to be tested, which contains the correspondence between tasks and business data in the task set, based on the task set. The determination module is used to perform anomaly detection based on the list to be detected and determine the abnormal job identification result of the batch job; The job set includes a target job set and a non-target job set. The generation module is specifically used to determine, based on the target job set, the first business data corresponding to each associated job included in the target job set, and generate a target data to be detected list containing each associated job and the first business data; and based on the non-target job set, determine the second business data corresponding to each non-associated job included in the non-target job set, and generate a non-target data to be detected list containing each non-associated job and the second business data. The determining module is specifically used to perform intersection processing on the first business data contained in the target data to be detected list and the second business data contained in the non-target data to be detected list to obtain a first data set to be detected; determine the business data contained in the first data set to be detected as abnormal business data, and determine the job corresponding to the abnormal business data as an abnormal job, and generate an abnormal job identification result. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5 when the computer program is executed by the processor. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.
7. A computer-readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.