Abnormal recognition method and device, electronic equipment and data processing system

Abstract

The invention discloses an exception recognition method and device, electronic equipment and a data processing system. Relates to the field of distributed stream processing, and comprises the following steps: in a stream processing framework, obtaining operation data generated by a target batch job in a target business system under the condition of processing business data, the operation data comprising a job name, operation end time and a business time range, the service time range represents a time range corresponding to the service data processed by the target batch job; based on the job names, standard time data corresponding to the target batch jobs are read in a database through a stream processing framework, and the standard time data comprise the relation between standard dates and service time ranges and standard moments; and based on the operation data and the standard time data, determining an abnormal identification result of the target batch job. Through the method and the device, the problems of poor real-time performance and low accuracy of batch operation aging detection in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of distributed stream processing, and in particular to an anomaly identification method, device, electronic device, and data processing system. Background Art

[0002] In today's data-driven enterprises and institutions, batch jobs are the core components for processing large amounts of data and generating key business information. Their runtime efficiency is directly related to business continuity and data accuracy. For example, in business processing, batch jobs may be responsible for tasks such as clearing, reconciliation, and risk assessment. These tasks need to be completed within a specified time to ensure the smooth operation of the next day's business.

[0003] However, the existing batch job timeliness monitoring technology has obvious deficiencies in terms of real-time performance and efficiency. Usually, the monitoring system will regularly read the operation status and SLA (service level agreement) data of the job from the relational database, and associate and compare them through scripts to determine whether the job is completed within the specified time. This method not only has poor real-time performance and cannot intervene in the first time when problems occur in the job, but also has high association complexity and is prone to errors. Especially when the number of batch jobs and the amount of data increase, this scheduled data processing and monitoring method will become slower and slower, affecting the efficiency and accuracy of monitoring.

[0004] Currently, no effective solution has been proposed to address the problems of poor real-time performance and low accuracy of batch job timeliness detection in related technologies. Summary of the invention

[0005] The embodiments of the present invention provide an abnormality identification method, device, electronic device and data processing system to at least solve the technical problems of poor real-time performance and low accuracy of batch job timeliness detection in the related art.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for identifying anomalies is provided. The method comprises: in a stream processing framework, obtaining the operation data generated by a target batch job in a target business system when processing business data, wherein the operation data comprises the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data comprises at least one of the following: conversion, calculation, classification, aggregation, and modeling; based on the job name, using the stream processing framework to read the standard time data corresponding to the target batch job in the database, wherein the standard time data comprises the relationship between the standard date and the business time range, and the standard time; based on the operation data and the standard time data, determining the abnormality identification result of the target batch job.

[0007] Optionally, the stream processing framework is a Flink fast distributed stream processing framework.

[0008] Optionally, based on the job name, a stream processing framework is used to read standard time data corresponding to a target batch job in a database, wherein the standard time data includes a relationship between a standard date and a business time range, and before a standard time, including: based on a user end of the target business system, obtaining standard time data corresponding to each of N batch jobs in the target business system, wherein the N batch jobs include the target batch job, and N is an integer greater than 1; in the stream processing framework, saving the standard time data corresponding to each of the N batch jobs in a database; in the stream processing framework, creating a data source corresponding to the database, and saving the address of the database in the data source.

[0009] Optionally, based on the operation data and the standard time data, an abnormality identification result of the target batch job is determined, including: determining the target standard date based on the business time range and the relationship between the standard date and the business time range; determining the target time range based on the target standard date and the standard time; judging whether the operation end time is outside the target time range; and determining that the abnormality identification result is that an abnormality exists when the operation end time is outside the target time range.

[0010] Optionally, when the abnormality identification result shows that an abnormality exists, a warning signal is sent to the user end of the target business system, and the operating data, standard time data and abnormality identification results are saved in a database; the operating data, standard time data and abnormality identification results are displayed on a preset screen.

[0011] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing system is provided. The system applies any one of the above-mentioned exception identification methods, including: a stream processing framework, which is used to obtain the operation data generated by the target batch job in the target business system when processing business data, read the standard time data corresponding to the target batch job, and determine the exception identification result of the target batch job; a message system, which is used to store the operation data; and a database, which is used to store the standard time data and the exception identification result.

[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an abnormality identification device is provided. The device includes: an acquisition module, which is used to acquire the operation data generated by the target batch job in the target business system when processing business data in the stream processing framework, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; a reading module, which is used to read the standard time data corresponding to the target batch job in the database based on the job name using the stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range and the standard time; a determination module, which is used to determine the abnormality identification result of the target batch job based on the operation data and the standard time data.

[0013] Optionally, the stream processing framework in the above device is a Flink fast distributed stream processing framework.

[0014] Optionally, the reading module also includes: an acquisition unit, a saving unit and a creation unit, wherein the acquisition unit is used to acquire the standard time data corresponding to each of N batch jobs in the target business system based on the user end of the target business system, wherein the N batch jobs include the target batch job, and N is an integer greater than 1; the saving unit is used to save the standard time data corresponding to each of the N batch jobs in a database in a stream processing framework; and the creation unit is used to create a data source corresponding to the database in the stream processing framework, and save the address of the database in the data source.

[0015] Optionally, the determination module also includes: a first determination unit, a second determination unit, a judgment unit and a third determination unit, wherein the first determination unit is used to determine the target standard date based on the business time range and the relationship between the standard date and the business time range; the second determination unit is used to determine the target time range based on the target standard date and the standard time; the judgment unit is used to judge whether the operation end time is outside the target time range; the third determination unit is used to determine that the abnormality identification result is an abnormality when the operation end time is outside the target time range.

[0016] Optionally, the device also includes: an early warning unit and a display unit, wherein the early warning unit is used to send an early warning signal to the user end of the target business system when the abnormality identification result is an abnormality, and save the operating data, standard time data and abnormality identification results in a database; the display unit is used to display the operating data, standard time data and abnormality identification results on a preset screen.

[0017] In an embodiment of the present invention, an anomaly identification method is adopted. In a stream processing framework, operation data generated by a target batch job in a target business system when processing business data is obtained, wherein the operation data includes a job name, an operation end time, and a business time range, wherein the business time range represents a time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; based on the job name, the standard time data corresponding to the target batch job is read in a database using the stream processing framework, wherein the standard time data includes a relationship between a standard date and a business time range, and a standard moment; based on the operation data and the standard time data, an anomaly identification result of the target batch job is determined, thereby achieving the purpose of real-time monitoring of the running status of the batch jobs of the target business system, thereby achieving the technical effect of improving the real-time and accuracy of batch job running time anomaly detection, and further solving the technical problems of poor real-time and low accuracy of batch job timeliness detection in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a computer terminal for implementing an abnormality identification method is shown;

[0020] Figure 2 is a flowchart of an abnormality identification method provided according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of performing associated reading based on a job name according to an optional embodiment of the present application;

[0022] Figure 4 is a flow chart of a real-time monitoring method based on an abnormality identification method provided according to an optional embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an abnormality identification device provided according to an embodiment of the present application;

[0024] Figure 6 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0028] Batch processing refers to the ability to run according to plan and access a large amount of data in a centralized manner without user interaction or when resources permit, to complete the expected processing or generate corresponding reports. Batch processing is suitable for commonly used programs that require little user interaction. They are mainly run at a planned time or in a centralized manner when needed.

[0029] Batch programs are used to implement specific batch processing functions and implement a single function (for example: a stored procedure or an executable program).

[0030] A batch job refers to a set of batch programs that form a set of functions that are closely connected based on logical dependencies. Generally, the internal process of a batch job is fixed and cannot be adjusted based on user or date selections.

[0031] Batch date: the date of the data used in batch processing. Usually different data are used for different dates.

[0032] Batch batches, on each batch date, the batch job can be executed only once or multiple times, each time is a batch.

[0033] Batch job SLA, or Service Level Agreement, specifically refers to the completion time set for batch jobs in the business process. If a batch job is not completed before the specified time, it may affect the acquisition of business data.

[0034] Batch job timeliness monitoring monitors the scheduled operation of batch jobs, filters out jobs that are not completed before the specified time, and issues alarms and emergency processing.

[0035] Flink is a distributed stream processing framework used to process large-scale data streams and perform data analysis and processing in real time.

[0036] MySQL, a relational database management system, is used to store and manage structured data.

[0037] It should be noted that the collected information and data involved in this application (including but not limited to data for display, data for analysis, etc.) are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions, providing users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0038] Example 1

[0039] According to an embodiment of the present application, an embodiment of an abnormality identification method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an abnormality identification method. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0041] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the abnormal identification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned abnormal identification method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0045] Under the above operating environment, this application provides Figure 2 The anomaly identification method shown. Figure 2 It is a flowchart of an abnormality identification method provided according to an embodiment of the present application.

[0046] Step S201, in the stream processing framework, obtain the operation data generated by the target batch job in the target business system when processing the business data, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling.

[0047] In this step, the stream processing framework can be used to monitor and process the target batch job running status in the target business system in real time. The stream processing framework can efficiently process the continuously arriving data stream, so that the execution of batch jobs can be tracked and analyzed in real time. For example, the batch job running data can be stored in Kafka and generated in real time by an external batch job running system. You can use Flink's Kafka data source function to create a Kafka data source or virtual table, set to the Kafka cluster address where the batch job running data is stored, and then Flink can obtain the newly added data from the data source in real time. The running data can include the execution status of the job, such as the job name, the start time and the end time of the operation, and the time range corresponding to the business data processed by the job, that is, the business time range, which defines the time period for the batch job to process the data, which can be a range or a batch date with a deadline. For example, one possible field of operation data is: job name, such as ABCD_EFG123; batch date (business time range), 2024-05-19; batch date batch number, 5 / 60, 1 / 1; operation start time, 2024-05-20 8:00:00; operation end time, 2024-05-20 9:00:00.

[0048] Step S202: Based on the job name, the standard time data corresponding to the target batch job is read from the database using the stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range, and the standard time.

[0049] In this step, the stream processing framework can not only be used to monitor the running data of the batch job in real time, but also to dynamically obtain the standard time data (SLA data) of the batch job to determine whether the job is completed on time. Specifically, the corresponding standard time data can be read from the database based on the job name. These data include the relationship between the standard date of the batch job and the business time range, as well as the standard time for the job, that is, the time point when the job should be completed. Figure 3 is a schematic diagram of performing associated reading based on a job name according to an optional embodiment of the present application, such as Figure 3 As shown, the job name is used to uniquely identify each batch job to ensure the accuracy of subsequent reading and comparison. The standard time data may include the following fields: job name, such as ABCD_EFG123; standard date, T+1 (T is the deadline of the business time range); standard time, 10:00:00.

[0050] Step S203: determining an abnormality recognition result of the target batch job based on the operation data and the standard time data.

[0051] In this step, once the running data and standard time data are associated, the stream processing framework can perform real-time detection of the completion status of each job. Specifically, it checks whether the running end time of each job exceeds the target time range defined in its SLA. If the running end time of the job is found to exceed the target time range, the stream processing framework will immediately generate an anomaly identification result. The result can contain detailed information about the job, such as the job name, business practice scope, batch number, running start time, running end time, and the duration of the timeout. For example, one possible field is: job name, such as ABCD_EFG123; business time range, 2024-05-19; batch day batch number, 5 / 60, 1 / 1; running start time, 2024-05-20 8:00:00; running end time, 2024-05-20 9:00:00; standard date, T+1 (T is the end date of the business time range); standard time, 10:00:00.

[0052] Optionally, in the exception identification method provided in an embodiment of the present application, the stream processing framework is a Flink fast distributed stream processing framework.

[0053] The stream processing framework is a technical means for real-time processing and analysis of large amounts of data streams, and specifically can be the Flink fast distributed stream processing framework. In this embodiment, Flink's real-time processing characteristics enable the running data of batch jobs to be analyzed instantly. Once it is found that the job is not completed within the standard time, an alarm can be triggered immediately, reducing delays and improving the abnormal response speed. Flink's high throughput and low-latency processing capabilities enable the running data of a large number of batch jobs to be processed quickly, improving the performance of the entire monitoring system. Through Flink's window and state management mechanism, the association process of batch job running data and SLA data can be simplified, avoiding the complex script writing and maintenance in traditional methods. Flink's state management and fault-tolerant mechanisms ensure that even if a failure occurs during data processing, it can be restored to the correct state to continue processing, improving the stability and reliability of the monitoring system.

[0054] Optionally, in the exception identification method provided in the embodiment of the present application, based on the job name, the standard time data corresponding to the target batch job is read in the database using a stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range, and before the standard time, including: based on the user end of the target business system, obtaining the standard time data corresponding to each of N batch jobs in the target business system, wherein the N batch jobs include the target batch job, and N is an integer greater than 1; in the stream processing framework, saving the standard time data corresponding to each of the N batch jobs in the database; in the stream processing framework, creating a data source corresponding to the database, and saving the address of the database in the data source.

[0055] First, the standard time data (SLA data) corresponding to each of the N batch jobs can be obtained from the target business system. These SLA data contain the target time range for each batch job to be completed, which is the key benchmark for judging whether the batch job is completed on time. Next, the standard time data of the N batch jobs obtained can be stored in the database. The database design should be able to efficiently store and retrieve this data, which usually includes fields such as job name and standard time. Storing SLA data in the database facilitates data query and association in the subsequent stream processing framework. In addition, in the stream processing framework Flink, a data source corresponding to the database can be created and the address of the database can be configured in the data source. Through this configuration, Flink can query and obtain the SLA data in the database in real time and compare it with the real-time job running data. By storing the batch job SLA data in the target business system in the database and creating the corresponding data source for real-time query and processing, the real-time and accuracy of batch job timeliness monitoring and the overall flexibility and scalability of the system can be significantly improved, providing a solid technical foundation for efficient and reliable batch job management.

[0056] Optionally, in the anomaly identification method provided in the embodiment of the present application, an anomaly identification result of the target batch job is determined based on the operation data and the standard time data, including: determining the target standard date based on the business time range and the relationship between the standard date and the business time range; determining the target time range based on the target standard date and the standard time; judging whether the operation end time is outside the target time range; and determining that the anomaly identification result is that an anomaly exists when the operation end time is outside the target time range.

[0057] Based on the business time range and the relationship between the standard date and the business time range (for example, the standard date is T+1 day of the business time range), the stream processing framework can determine the target standard date. Next, based on the target standard date and the standard time set in the SLA of the batch job (that is, the job should be completed before a certain time point on the target standard date), the system determines the target time range, which refers to the time window in which the batch job should be completed. The stream processing framework can obtain and process batch job operation data in real time, including the job's run end time. The system compares the run end time with the target time range. If the end time is outside the target time range, that is, the job completion time exceeds the completion time specified in the SLA, this will be identified as an exception.

[0058] For example, the 1 / 1 batch of the batch job ABCD_EFG123 in the business time range 2024-05-19 starts running at 2024-05-20 8:00:00 and ends at 2024-05-20 9:00:00. In the batch job SLA, the standard date of the job is 10:00:00 on T+1 day, where T is the deadline of the business time range. Then the completion time of the batch job ABCD_EFG123 in the business time range 2024-05-19 should be before 2024-05-20 10:00:00. After comparison, the job is executed and completed within the specified time.

[0059] Optionally, in the anomaly identification method provided in the embodiment of the present application, when the anomaly identification result is that an anomaly exists, a warning signal is sent to the user end of the target business system, and the operating data, standard time data and the anomaly identification result are saved in a database; the operating data, standard time data and the anomaly identification result are displayed on a preset screen.

[0060] When it is determined that the end time of the batch job exceeds the target time range specified in its SLA, that is, when a timeout anomaly is identified, a warning signal can be sent to the user end of the target business system through a series of steps, and the relevant data can be saved and displayed to ensure that the business personnel can obtain the abnormal information and handle it in time. Specifically, when the abnormality identification result shows that there is an abnormality, the system can immediately send a warning signal to the user end of the target business system. For example, an email, text message or push notification can be sent to the business personnel to ensure that they can immediately understand the details of the timed job. The identified abnormal information, including operation data, standard time data and abnormality identification results, can be uniformly saved in the database, providing data support for subsequent fault analysis, problem tracing and data reporting. The abnormality identification results and related data can not only be saved, but also be visualized on the preset screen so that the business personnel can intuitively see the execution status of the batch job.

[0061] Therefore, the process steps of a real-time monitoring method based on the abnormality identification method provided in the embodiment of the present application may be: data acquisition, data association and data storage. Figure 4 is a flow chart of a real-time monitoring method based on an abnormality identification method provided according to an optional embodiment of the present application, such as Figure 4As shown in Figure 1, data acquisition is the first step of the real-time monitoring method and the basis of data stream processing. At this stage, the stream processing framework obtains the running data of the target batch job from the message system in real time. At the same time, the stream processing framework also needs to read the standard time data (SLA data) of the batch job from the database (such as MySQL). Data association is the core step of the real-time monitoring method. Its purpose is to compare the acquired running data with the standard time data to determine whether the job is completed on time. In Flink, functions such as window functions and state management can be used to efficiently associate the real-time acquired running data with SLA data. Data storage is the last step of the real-time monitoring process, which is used to record and save the anomaly identification results. Once a timeout anomaly is detected in the job, the stream processing framework will store the relevant running data, standard time data, and anomaly identification results in the database.

[0062] The anomaly identification method provided by the embodiment of the present application obtains the operation data generated by the target batch job in the target business system when processing business data in the stream processing framework, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; based on the job name, the standard time data corresponding to the target batch job is read in the database using the stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range, and the standard time; based on the operation data and the standard time data, the anomaly identification result of the target batch job is determined, so as to achieve the purpose of real-time monitoring of the batch job operation status of the target business system, thereby realizing the technical effect of improving the real-time and accuracy of batch job operation time anomaly detection, and further solving the technical problems of poor real-time and low accuracy of batch job time detection in related technologies.

[0063] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0064] Example 2

[0065] The embodiment of the present application also provides a data processing system. It should be noted that the data processing system of the embodiment of the present application can apply the abnormality identification method provided in the embodiment of the present application. The data processing system provided in the embodiment of the present application is introduced below.

[0066] According to an embodiment of the present application, a system for implementing the above-mentioned abnormality identification method is also provided, and the system includes:

[0067] The stream processing framework is used to obtain the operation data generated by the target batch job in the target business system when processing business data, read the standard time data corresponding to the target batch job, and determine the abnormality recognition result of the target batch job.

[0068] Message system, used to store operation data.

[0069] Database, used to store standard time data and anomaly recognition results.

[0070] The stream processing framework, messaging system, and database work together to achieve real-time acquisition of operational data generated by target batch jobs when processing business data, reading of standard time data, and determination and storage of anomaly identification results.

[0071] The stream processing framework undertakes the core data processing and analysis tasks. It can obtain the running data of the target batch job from the message system (such as Kafka) in real time. At the same time, the stream processing framework can also read the standard time data corresponding to the target batch job from the database, that is, the completion date and time specified in the SLA. Based on this data, the stream processing framework can immediately identify anomalies in the case of job timeouts through real-time data processing and comparison, generate anomaly identification results, and store these results in the database for subsequent analysis and monitoring.

[0072] The message system serves as a transmission and storage platform for operation data. When batch jobs in the target business system are executed, operation data can be generated in real time and stored persistently through Kafka's message queue. As a high-throughput, low-latency message system, Kafka can process large-scale real-time data streams, ensuring that the stream processing framework can obtain the operation data of the job in a timely and complete manner, providing a solid data foundation for real-time monitoring.

[0073] The database, such as MySQL or PostgreSQL, is responsible for storing operation data, standard time data, and anomaly identification results. The database design needs to ensure efficient storage and retrieval of data so that the stream processing framework can quickly read the standard time data for comparison, and it is also convenient for business personnel to query the operation status and anomaly records of the job.

[0074] Example 3

[0075] The embodiment of the present application also provides an abnormality identification device. It should be noted that the abnormality identification device of the embodiment of the present application can be used to execute the abnormality identification method provided in the embodiment of the present application. The abnormality identification device provided in the embodiment of the present application is introduced below.

[0076] According to an embodiment of the present application, a device for implementing the above-mentioned abnormality identification method is also provided, such as Figure 5 As shown, the device comprises:

[0077] The acquisition module 51 is used to acquire the operation data generated by the target batch job in the target business system when processing business data in the stream processing framework, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling.

[0078] The reading module 52 is connected to the acquisition module 51 and is used to read the standard time data corresponding to the target batch job in the database based on the job name using the stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range and the standard time.

[0079] The determination module 53 is connected to the reading module 52 and is used to determine the abnormality recognition result of the target batch job based on the operation data and the standard time data.

[0080] The abnormality identification device provided in the embodiment of the present application is used, through the acquisition module 51, to obtain the operation data generated by the target batch job in the target business system when processing business data in the stream processing framework, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; the reading module 52 is used to read the standard time data corresponding to the target batch job in the database based on the job name using the stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range, and the standard time; the determination module 53 is used to determine the abnormality identification result of the target batch job based on the operation data and the standard time data, so as to achieve the purpose of real-time monitoring of the batch job operation status of the target business system, thereby realizing the technical effect of improving the real-time and accuracy of batch job operation time anomaly detection, and further solving the technical problems of poor real-time and low accuracy of batch job time detection in related technologies.

[0081] Optionally, in the anomaly identification device provided in an embodiment of the present application, the stream processing framework is a Flink fast distributed stream processing framework.

[0082] Optionally, in the anomaly identification device provided in the embodiment of the present application, the reading module 52 also includes: an acquisition unit, a saving unit and a creation unit, wherein the acquisition unit is used to acquire the standard time data corresponding to each of N batch jobs in the target business system based on the user end of the target business system, wherein the N batch jobs include the target batch job, and N is an integer greater than 1; the saving unit is used to save the standard time data corresponding to each of the N batch jobs in a database in a stream processing framework; the creation unit is used to create a data source corresponding to the database in the stream processing framework, and save the address of the database in the data source.

[0083] Optionally, in the abnormality identification device provided in the embodiment of the present application, the determination module 53 also includes: a first determination unit, a second determination unit, a judgment unit and a third determination unit, wherein the first determination unit is used to determine the target standard date based on the business time range and the relationship between the standard date and the business time range; the second determination unit is used to determine the target time range based on the target standard date and the standard time; the judgment unit is used to judge whether the operation end time is outside the target time range; the third determination unit is used to determine that the abnormality identification result is an abnormality when the operation end time is outside the target time range.

[0084] Optionally, the abnormality identification device provided in the embodiment of the present application also includes: an early warning unit and a display unit, wherein the early warning unit is used to send an early warning signal to the user end of the target business system when the abnormality identification result is an abnormality, and save the operating data, standard time data and abnormality identification results in a database; the display unit is used to display the operating data, standard time data and abnormality identification results on a preset screen.

[0085] It should be noted that the acquisition module 51, the reading module 52 and the determination module 53 correspond to steps S201 to S203 in Example 1, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules can also be run in the computer terminal 10 provided in Example 1 as part of the device.

[0086] Example 4

[0087] An embodiment of the present application may provide an electronic device, Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device may include: one or more ( Figure 6 (only one is shown) processor 602, memory 604, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0088] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0089] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: in the stream processing framework, obtain the operation data generated by the target batch job in the target business system when processing the business data, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; based on the job name, use the stream processing framework to read the standard time data corresponding to the target batch job in the database, wherein the standard time data includes the relationship between the standard date and the business time range, and the standard time; based on the operation data and the standard time data, determine the abnormality recognition result of the target batch job.

[0090] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: The stream processing framework is the Flink fast distributed stream processing framework.

[0091] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: based on the job name, use the stream processing framework to read the standard time data corresponding to the target batch job in the database, wherein the standard time data includes the relationship between the standard date and the business time range, and before the standard time, including: based on the user end of the target business system, obtain the standard time data corresponding to each of N batch jobs in the target business system, wherein the N batch jobs include the target batch job, and N is an integer greater than 1; in the stream processing framework, save the standard time data corresponding to each of the N batch jobs in the database; in the stream processing framework, create a data source corresponding to the database, and save the address of the database in the data source.

[0092] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the abnormality identification result of the target batch job based on the operation data and the standard time data, including: determine the target standard date based on the business time range and the relationship between the standard date and the business time range; determine the target time range based on the target standard date and the standard time; determine whether the operation end time is outside the target time range; if the operation end time is outside the target time range, determine the abnormality identification result as the existence of an abnormality.

[0093] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: when the abnormality identification result is that an abnormality exists, send an early warning signal to the user end of the target business system, and save the operation data, standard time data and abnormality identification results in the database; display the operation data, standard time data and abnormality identification results on a preset screen.

[0094] According to an embodiment of the present application, an anomaly identification method is provided. By acquiring the operation data generated by the target batch job in the target business system when processing business data in a stream processing framework, wherein the operation data includes the job name, the operation end time and the business time range, the business time range represents the time range corresponding to the business data processed by the target batch job, and the operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; based on the job name, the standard time data corresponding to the target batch job is read in the database using the stream processing framework, wherein the standard time data includes the relationship between the standard date and the business time range and the standard time; based on the operation data and the standard time data, the anomaly identification result of the target batch job is determined, so as to achieve the purpose of real-time monitoring of the running status of the batch job of the target business system, thereby achieving the technical effect of improving the real-time and accuracy of the batch job running time anomaly detection, and thus solving the technical problems of poor real-time and low accuracy of batch job time detection in the related technology.

[0095] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations are shown.

[0096] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0097] Example 5

[0098] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the abnormality identification method provided in the first embodiment.

[0099] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0100] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the anomaly identification method.

[0101] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0102] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0107] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for identifying anomalies, characterized in that: include: In a stream processing framework, operation data generated by a target batch job in a target business system when processing business data is obtained, wherein the operation data includes a job name, an operation end time, and a business time range, the business time range represents a time range corresponding to the business data processed by the target batch job, and an operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; Based on the job name, using the stream processing framework to read the standard time data corresponding to the target batch job in the database, wherein the standard time data includes the relationship between the standard date and the business time range, and the standard time; Based on the operation data and the standard time data, an abnormality identification result of the target batch job is determined.

2. The method according to claim 1, characterized in that The stream processing framework is the Flink fast distributed stream processing framework.

3. The method according to claim 1, characterized in that In the step of reading the standard time data corresponding to the target batch job in the database by using the stream processing framework based on the job name, wherein the standard time data includes the relationship between the standard date and the business time range and before the standard time, including: Based on the user end of the target business system, obtain standard time data corresponding to each of N batch jobs in the target business system, wherein the N batch jobs include the target batch job, and N is an integer greater than 1; In the stream processing framework, the standard time data corresponding to each of the N batch jobs is stored in the database; In the stream processing framework, a data source corresponding to the database is created, and the address of the database is stored in the data source.

4. The method according to claim 1, characterized in that: The determining, based on the operation data and the standard time data, an abnormality recognition result of the target batch job includes: determining a target standard date based on the business time range and a relationship between the standard date and the business time range; Determining a target time range based on the target standard date and the standard time; Determining whether the operation end time is outside the target time range; When the operation end time is outside the target time range, the abnormality identification result is determined to be abnormal.

5. The method according to claim 1, characterized in that Also includes: When the abnormality identification result indicates that an abnormality exists, a warning signal is sent to a user end of the target business system, and the operation data, the standard time data and the abnormality identification result are stored in the database; The operating data, the standard time data and the abnormality identification result are displayed on a preset screen.

6. A data processing system, characterized in that: The method for identifying anomalies according to any one of claims 1 to 5 is applied, comprising: A stream processing framework, used to obtain operation data generated by a target batch job in a target business system when processing business data, read standard time data corresponding to the target batch job, and determine an abnormality recognition result of the target batch job; A message system, used for storing the operation data; A database is used to store the standard time data and the abnormality identification results.

7. An abnormality identification device, characterized in that: include: an acquisition module, configured to acquire, in a stream processing framework, operation data generated by a target batch job in a target business system when processing business data, wherein the operation data includes a job name, an operation end time, and a business time range, the business time range represents a time range corresponding to the business data processed by the target batch job, and an operation of processing the business data includes at least one of the following: conversion, calculation, classification, aggregation, and modeling; A reading module, configured to read the standard time data corresponding to the target batch job in the database by using the stream processing framework based on the job name, wherein the standard time data includes the relationship between the standard date and the business time range, and the standard time; A determination module is used to determine an abnormality recognition result of the target batch job based on the operation data and the standard time data.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the abnormality identification method according to any one of claims 1 to 5.

9. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the abnormality identification method described in any one of claims 1 to 5 when running.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the abnormality identification method described in any one of claims 1 to 5 are implemented.

Images