Data exception monitoring method and device, electronic equipment and storage medium
By aggregating and analyzing and matching the abnormal data and logs in data exception monitoring, and combining the display of abnormal logs, the problems of the causes of abnormalities in data exception monitoring in the prior art are solved, and more accurate abnormal monitoring and Real-time alarm.
Patent Information
- Application Number
- CN202411114354.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art relies heavily on a single exception log in data exception monitoring, which affects the accuracy of the cause of target exceptions and makes users unable to intuitively know the specific details.
By aggregating and analyzing the specified data after data processing, combining exception data and exception logs, the cause of the target exception is determined, and the associated target exception log is displayed on the monitoring page, including component and level matching analysis of the abnormal data. And store and display exception logs through message queues.
It improves the accuracy of the cause of target exceptions, enables users to intuitively understand the causes and details of data exceptions, reduces errors caused by abnormal log errors, and realizes real-time exception monitoring and sending alarm information.
Smart Images

Figure CN119025319A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a method, device, electronic device and storage medium for monitoring data anomalies. Background Art
[0002] Currently, the monitoring of data anomalies only relies on the anomaly logs recorded during the monitoring process to determine the cause of the target anomaly. Because it is highly dependent on a single anomaly log, if there is an error in the anomaly log, it will directly affect the accuracy of the cause of the target anomaly.
[0003] At the same time, during the data anomaly monitoring process, in order to facilitate users to intuitively understand the reasons for data anomalies, only the target anomaly reasons will be displayed on the monitoring page, making it impossible for users to know the specific details. Summary of the invention
[0004] In view of the above problems, the present application provides a data anomaly monitoring method, device, electronic device and storage medium to solve the problems existing in the prior art.
[0005] According to one aspect of the present application, a method for monitoring data anomalies is provided, the monitoring method comprising: performing aggregation analysis on designated data after data processing to determine abnormal data in the designated data; determining a target anomaly cause based on the abnormal data and an abnormal log; wherein the abnormal log is a log recorded when data processing is abnormal during data processing; the target anomaly cause and the target anomaly log are displayed on a monitoring page; wherein the target anomaly log is a log in the anomaly log that is associated with the target anomaly cause.
[0006] In an optional manner, determining the target abnormality cause based on the abnormal data and the abnormal log further includes: taking the component corresponding to the abnormal data as the target component, and determining the abnormality log of the target abnormality level corresponding to the target component from the abnormality log; determining the target abnormality cause based on the abnormality log of the target abnormality level corresponding to the target component and the abnormal data.
[0007] In an optional manner, determining the target exception cause based on the exception log of the target exception level corresponding to the target component and the exception data further includes: obtaining the exception code in the exception log of the target exception level corresponding to the target component; matching the exception code with a preset exception code, and merging and analyzing the exception causes corresponding to the successfully matched preset exception codes to determine the target exception cause.
[0008] In an optional manner, the aggregation analysis of the designated data after data processing to determine the abnormal data in the designated data further includes: performing a variance operation on the designated data of the same type after data processing to obtain variance values corresponding to multiple different types of designated data; performing a difference operation on the designated data of each type and their corresponding variance values, and treating the designated data whose difference is greater than a preset difference as abnormal data.
[0009] In an optional manner, the aggregation analysis of the designated data after data processing to determine the abnormal data in the designated data further includes: traversing the designated data after data processing, and taking the traversed designated data as the target designated data; obtaining all target data on the target data processing chain corresponding to the target designated data, and matching the value of each target data with the corresponding preset value to obtain a matching rate corresponding to the target data processing chain; if the matching rate is less than the preset matching rate, taking the target designated data as abnormal data.
[0010] In an optional manner, the monitoring method also includes: if the number of the target abnormality reasons reaches a preset number, or the current moment is within a preset alarm period, an alarm message is generated based on the target abnormality reason and the target abnormality log to be sent to different user terminals.
[0011] In an optional manner, the monitoring method also includes: recording the initial exception log generated during the data processing process, and determining the exception log of the target exception level in the initial exception log; storing the exception log of the target exception level in the log database through the message queue, and displaying the exception log in the log database on the monitoring page.
[0012] According to another aspect of the present application, a data anomaly monitoring device is provided, the monitoring device comprising: an analysis and determination module, used to perform aggregate analysis on designated data after data processing, and determine abnormal data in the designated data; a cause determination module, used to determine a target anomaly cause based on the abnormal data and an abnormal log; wherein the abnormal log is a log recorded when data processing is abnormal during data processing; a display module, used to display the target anomaly cause and the target anomaly log on a monitoring page; wherein the target anomaly log is a log in the anomaly log that is associated with the target anomaly cause.
[0013] According to one aspect of the present application, an electronic device is provided, including: a controller; and a memory for storing one or more programs, wherein when the one or more programs are executed by the controller, the above-mentioned monitoring method is executed.
[0014] According to one aspect of the present application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above-mentioned monitoring method.
[0015] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned monitoring method.
[0016] This application combines the abnormal log recorded in the data abnormality monitoring process with the specified data after data processing to determine the cause of the target abnormality. Among them, the data recorded in the data processing process and the data after data processing are combined, and the relevant data of the data processing process are comprehensively considered, so that the determined cause of the target abnormality is more accurate. At the same time, the target abnormality cause and the target abnormality log are also displayed on the monitoring page, so that the user can intuitively know the cause of the target abnormality and related detailed data.
[0017] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0019] Figure 1 It is a flowchart of a method for monitoring data anomalies shown in an exemplary embodiment of the present application.
[0020] Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for monitoring data anomalies.
[0021] Figure 3 It is a structural diagram of a data processing chain shown in an exemplary embodiment of the present application.
[0022] Figure 4 It is a schematic diagram of an application scenario of the data anomaly monitoring method of the present application.
[0023] Figure 5 It is a structural diagram of a data anomaly monitoring device shown in an exemplary embodiment of the present application.
[0024] Figure 6 It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0025] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0027] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0028] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0029] Existing monitoring of data anomalies only relies on the anomaly logs recorded during the monitoring process to determine the cause of the target anomaly. Because it is highly dependent on a single anomaly log, if there is an error in the anomaly log, it will directly affect the accuracy of the cause of the target anomaly.
[0030] At the same time, in the process of data anomaly monitoring, the anomaly log only records the anomaly of a single data, and does not consider the mutual influence between different stages or different types of data, which will also affect the accuracy of the determined target anomaly cause. For example, the data is normal during the monitoring process, but when they are aggregated and analyzed, corresponding errors will occur. If only the anomaly log is monitored, similar errors cannot be detected.
[0031] In addition, in order to help users intuitively understand the reasons for data anomalies, only the reasons for the target anomalies will be displayed on the monitoring page, making it impossible for users to know the specific details.
[0032] To this end, one aspect of the present application provides a method for monitoring data anomalies. Figure 1 , Figure 1 1 is a flow chart of a data anomaly monitoring method shown in an exemplary embodiment of the present application. The monitoring method at least includes S110 to S130, which are described in detail as follows:
[0033] S110: performing aggregation analysis on the designated data after data processing to determine abnormal data in the designated data.
[0034] The designated data is data that does not represent abnormal conditions during data processing. The designated data represents the processed data, that is, the data within a defined range. There is a certain connection between the designated data. For example, the designated data of this embodiment can be data within a specified time range, or data of a specified type, or data related to a specified component. This embodiment does not limit its specific meaning, and it can be adaptively adjusted according to actual conditions.
[0035] Aggregate analysis is a data analysis performed on specified data, that is, the corresponding specified data is analyzed as a whole object, and the impact of each data on the analysis result and the relationship between each data need to be considered, so as to accurately determine the abnormal data. Aggregate analysis includes but is not limited to variance analysis, data processing chain analysis, etc.
[0036] S120: Determine the target abnormality cause according to the abnormal data and the abnormal log; wherein the abnormal log is a log recorded when data processing is abnormal during the data processing process.
[0037] In this embodiment, the number of abnormal data and abnormal logs can be multiple. The abnormal log generally records the corresponding abnormal occurrence time, abnormal code, abnormal component, etc.; the abnormal data corresponds to the corresponding abnormal component, abnormal code, etc. According to the common connection between the abnormal data and the abnormal log, the corresponding target abnormal cause can be determined. For example, based on 3 abnormal data representing ECU abnormalities, and the log representing the ECU abnormality recorded in the abnormal log, the target abnormal cause can be analyzed and determined.
[0038] S130: Displaying the target exception cause and the target exception log on the monitoring page; wherein the target exception log is a log in the exception log that is associated with the target exception cause.
[0039] When displaying the target exception reason, the target exception log is highlighted by marking it in yellow or bold, so that the user can intuitively know the target exception reason through the monitoring page, and can quickly obtain the associated target exception log. This embodiment can display the exception log in real time during the data processing process, and the specific examples are as follows:
[0040] Record the initial exception logs generated during data processing, and determine the exception logs of the target exception level in the initial exception logs. Filter the initial exception logs by exception level to filter out the exception logs that meet the exception level requirements.
[0041] The abnormal logs of the target abnormal level are stored in the log database through the message queue, and the abnormal logs in the log database are displayed on the monitoring page. Among them, the message queue can buffer a large number of abnormal logs to avoid a large number of abnormal logs being stored in the log database at the same time, which may cause the system to crash. Especially in the big data processing scenario, the amount of data processed at each moment is huge, and the number of abnormal logs is also large. The message queue can ensure the normal operation of log storage, and at the same time will not affect the data processing capability of the real-time program.
[0042] The exception logs in the log database can be displayed in real time on the monitoring page. It is used to specifically store exception logs, store exception logs and processed data in different areas, and quickly extract different data from the corresponding database.
[0043] In some embodiments, when certain preset conditions are met, corresponding warning information may be sent to different user terminals to enable users to quickly know data anomalies. Specific examples are as follows:
[0044] If the number of target abnormal reasons reaches a preset number, or the current moment is within a preset alarm period, an alarm message is generated based on the target abnormal reason and the target abnormal log to be sent to different user terminals.
[0045] In order to avoid accidental errors, a preset number is set to avoid false alarms caused by accidental errors. In addition, the alarm operation can also be performed through the time ruler, that is, the current moment is matched with the time interval represented by the preset alarm cycle. If the current moment is in the time interval, the alarm operation can be performed.
[0046] The alarm information is to refine the information of the target abnormal cause and the target abnormal log, and simplify it into alarm information that conforms to the cognition of ordinary users, so that users can quickly understand the abnormal cause and corresponding details through the alarm information, such as the time when the abnormality occurred, the abnormal component, etc.
[0047] This embodiment combines the abnormal log recorded in the data abnormality monitoring process with the specified data after data processing to determine the target abnormality cause. Among them, the data recorded in the data processing process and the data after data processing are combined, and the relevant data of the data processing process are comprehensively considered, so that the determined target abnormality cause is more accurate. At the same time, the target abnormality cause and the target abnormality log are also displayed on the monitoring page, so that the user can intuitively know the target abnormality cause and related detailed data.
[0048] In another exemplary embodiment of the present application, how to determine the cause of the target abnormality based on the abnormal data and abnormal log is described in detail. Figure 2 , Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for monitoring data anomalies. The monitoring method is as follows Figure 1 The S120 shown further includes S210 to S220, which are described in detail as follows:
[0049] S210: Taking the component corresponding to the abnormal data as the target component, and determining the abnormality log of the target abnormality level corresponding to the target component from the abnormality log.
[0050] Abnormal data indicates abnormal conditions of corresponding components, and the abnormal codes in the abnormal data correspond to the corresponding components.
[0051] The exception log represents logs of multiple different exception levels, generally including debug level, information level, warning level, and error level. The target exception levels in this embodiment are warning level and error level.
[0052] This embodiment screens the abnormal logs for the target component, and determines the abnormal logs of the target abnormal level corresponding to the corresponding target component in combination with the classified abnormal levels.
[0053] S220: Determine the cause of the target abnormality according to the abnormality log and abnormality data of the target abnormality level corresponding to the target component.
[0054] Exemplarily, the abnormal code in the abnormal log of the target abnormal level is extracted to determine the corresponding abnormal accident and cause, and combined with the abnormal data determined after data processing, the cause of the target abnormality can be accurately determined.
[0055] Specifically, the abnormality code in the abnormality log of the target abnormality level corresponding to the target component is obtained; the abnormality code is matched with the preset abnormality code, and the abnormality reasons corresponding to the successfully matched preset abnormality codes are combined and analyzed to determine the target abnormality reason. Each preset abnormality code corresponds to a corresponding abnormality reason.
[0056] For example, the preset exception codes that are matched successfully include 0x001 and 0x002. The preset exception code 0x001 corresponds to ECU data calculation error, and 0x002 corresponds to ECU data missing. The target exception cause obtained by combining them for analysis is ECU data processing exception.
[0057] This embodiment screens the abnormal logs by the target component corresponding to the abnormal data, thereby determining the abnormal logs related to the target component, and accurately determining the target abnormal cause, that is, the abnormal cause of the target component, based on the abnormal data and the details recorded in the abnormal logs.
[0058] In another exemplary embodiment of the present application, it is described in detail how to perform aggregate analysis on the specified data after data processing to determine the abnormal data in the specified data, that is, in the following example: Figure 1 The S110 shown further includes S1101 to S1102, which are described in detail as follows:
[0059] S1101: performing variance calculation on designated data of the same type after data processing to obtain variance values corresponding to designated data of different types.
[0060] Exemplarily, an average value operation is performed on the specified data of the controller type, and then the variance value of the specified data of this type is calculated, and so on, the variance value corresponding to the specified data of each type is calculated.
[0061] S1102: Perform difference calculation on designated data of each type and their corresponding variance values, and regard designated data with a difference greater than a preset difference as abnormal data.
[0062] Exemplarily, for the specified data of the controller type, a difference operation is performed between each specified data and the variance value corresponding to the controller type, and each difference is compared with a preset difference. If it is greater than the preset difference, it indicates that there is a large deviation in the numerical value corresponding to the specified data, and it is regarded as abnormal data.
[0063] In another exemplary embodiment of the present application, another method of performing aggregate analysis on the specified data after data processing to determine abnormal data in the specified data is described in detail, that is, in the following example: Figure 1 The S110 shown further includes S1103 to S1105, which are described in detail as follows:
[0064] S1103: traverse the designated data after data processing, and use the traversed designated data as target designated data.
[0065] This embodiment adopts a traversal method to ensure that each designated data is processed in a targeted manner without omission, so as to prepare to determine abnormal data.
[0066] S1104: Acquire all target data on the target data processing chain corresponding to the target designated data, and match the value of each target data with the corresponding preset value to obtain a matching rate corresponding to the target data processing chain.
[0067] See also Figure 3 , Figure 3 It is a structural diagram of a data processing chain shown in an exemplary embodiment of the present application. Among them, the end data of each data processing chain is the designated data after data processing, and each data processing chain has multiple node data. This embodiment performs corresponding inspections on all data on each data processing chain, and obtains the matching rate of each data on the corresponding data processing chain by matching the data value with the corresponding preset value, so as to obtain the matching rate corresponding to the corresponding data processing chain, and then determine whether the corresponding designated data is abnormal data.
[0068] S1105: If the matching rate is less than the preset matching rate, the target designated data is regarded as abnormal data.
[0069] This embodiment does not limit the specific numerical value of the preset matching rate. If the matching rate of the data processing chain is greater than or equal to the preset matching rate, it means that the specified data after the data processing at its end is normal. Otherwise, it can be determined as abnormal data.
[0070] The present application can determine abnormal data in the specified data through different aggregation analysis methods. The aggregation analysis methods shown in the above exemplary embodiments can accurately determine abnormal data from different thinking levels.
[0071] In another exemplary embodiment of the present application, the application scenarios of the above-mentioned multiple monitoring methods are exemplarily described. Figure 4 , Figure 4 It is a schematic diagram of the application scenario of the data anomaly monitoring method of the present application. Among them, the big data real-time processing program 101 processes data of different types of data sources, and records corresponding abnormal logs in real time during the data processing process, outputs the abnormal data log stream to the abnormal data log pipeline 103 (generally a kafka message queue) through the log outputter 102, and then stores it in the log database 104, so as to display the corresponding abnormal log on the log abnormality monitoring dashboard 105.
[0072] At the same time, the data processed by the big data real-time processing program 101 is stored in the target business database 106, which contains all processed data, from which the corresponding abnormal data can be determined to be collected by the abnormal data timing collector 107. The abnormal data timing collector 107 can also collect the corresponding abnormal log from the log database 104, and determine the cause of the abnormality based on the abnormal data and the abnormal log, so as to summarize all the causes of the abnormality in the abnormal problem summary library 108. The abnormal problem summary library 108 displays the corresponding cause of the abnormality on the log abnormality monitoring dashboard 105, so that the cause of the abnormality and the related abnormal log are displayed synchronously, so that the user can understand the abnormal situation more intuitively.
[0073] If the number of exception causes in the exception problem summary library 108 reaches a preset number, or the current moment is within the preset alarm cycle, the alarm manager 109 is triggered to generate alarm information based on the target exception cause and the target exception log, and send it to different user terminals so that users can understand the abnormal situation of data processing in a timely manner through different user terminals.
[0074] This embodiment makes full use of the data in the log database 104 and the abnormal problem summary database 108 as the data source for abnormal monitoring, so that the abnormal log detailed data can be checked and the abnormal statistical data can be consulted. In addition, this embodiment can perform abnormal monitoring on the data in data processing and the data in the target business database data 106 after data processing, that is, it realizes comprehensive monitoring of all abnormal data.
[0075] Another aspect of the present application also provides a data anomaly monitoring device, such as Figure 5 As shown, Figure 5 1 is a schematic diagram of a data anomaly monitoring device according to an exemplary embodiment of the present application. The monitoring device 500 includes:
[0076] The analysis and determination module 510 is used to perform aggregation analysis on the designated data after data processing to determine abnormal data in the designated data.
[0077] The cause determination module 530 is used to determine the target abnormal cause according to the abnormal data and the abnormal log; wherein the abnormal log is a log recorded when the data processing is abnormal during the data processing process.
[0078] The display module 550 is used to display the target exception cause and the target exception log on the monitoring page; wherein the target exception log is a log in the exception log that is associated with the target exception cause.
[0079] In another exemplary embodiment, the cause determination module 530 further includes:
[0080] The log determination unit is used to take the component corresponding to the abnormal data as the target component, and determine the abnormal log of the target abnormal level corresponding to the target component from the abnormal log.
[0081] The cause determination unit is used to determine the target abnormality cause according to the abnormality log and abnormality data of the target abnormality level corresponding to the target component.
[0082] In another exemplary embodiment, the cause determination unit further includes:
[0083] The acquisition section is used to obtain the exception code in the exception log of the target exception level corresponding to the target component.
[0084] The matching section is used to match the exception code with the preset exception code, and to combine and analyze the exception causes corresponding to the successfully matched preset exception codes to determine the target exception cause.
[0085] In another exemplary embodiment, the analysis and determination module 510 further includes:
[0086] The variance calculation unit is used to perform variance calculation on the specified data of the same type after data processing, and obtain variance values corresponding to multiple specified data of different types.
[0087] The first determining unit is used to perform a difference operation between each type of designated data and the corresponding variance value, and regard the designated data whose difference is greater than a preset difference as abnormal data.
[0088] In another exemplary embodiment, the analysis and determination module 510 further includes:
[0089] The traversal unit is used to traverse the designated data after data processing, and use the traversed designated data as target designated data.
[0090] The acquisition matching unit is used to acquire all target data on the target data processing chain corresponding to the target designated data, and match the value of each target data with the corresponding preset value to obtain the matching rate corresponding to the target data processing chain.
[0091] The second determining unit is configured to regard the target designated data as abnormal data if the matching rate is less than a preset matching rate.
[0092] In another exemplary embodiment, the monitoring device 500 further includes:
[0093] The alarm module is used to generate alarm information based on the target abnormality cause and the target abnormality log to send to different user terminals if the number of target abnormality causes reaches a preset number or the current moment is within a preset alarm period.
[0094] In another exemplary embodiment, the monitoring device 500 further includes:
[0095] The recording module is used to record the initial exception log generated during the data processing process and determine the exception log of the target exception level in the initial exception log.
[0096] The storage and display module is used to store the abnormal logs of the target abnormal level into the log database through the message queue, and display the abnormal logs in the log database on the monitoring page.
[0097] The monitoring device of the present application combines the abnormal log recorded in the data abnormality monitoring process with the specified data after data processing to determine the cause of the target abnormality. Among them, the data recorded in the data processing process and the data after data processing are combined, and the relevant data of the data processing process are comprehensively considered, so that the determined cause of the target abnormality is more accurate. At the same time, the target abnormality cause and the target abnormality log are also displayed on the monitoring page, so that the user can intuitively know the cause of the target abnormality and related detailed data.
[0098] It should be noted that the monitoring device provided in the above embodiment and the monitoring method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.
[0099] Another aspect of the present application provides an electronic device, including: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the above-mentioned monitoring method is executed.
[0100] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of a computer system of an electronic device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.
[0101] It should be noted that Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0102] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the method in the above embodiment. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0103] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.
[0104] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.
[0105] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0106] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0107] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0108] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the monitoring method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.
[0109] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the monitoring method provided in each of the above embodiments.
[0110] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0111] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.
[0112] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
Claims
1. A method for monitoring data anomalies, characterized in that: The monitoring method comprises: Performing aggregate analysis on the designated data after data processing to determine abnormal data in the designated data; Determine the cause of the target abnormality according to the abnormal data and the abnormal log; wherein the abnormal log is a log recorded when data processing is abnormal during the data processing process; The target exception cause and the target exception log are displayed on a monitoring page; wherein the target exception log is a log in the exception log that is associated with the target exception cause.
2. The monitoring method according to claim 1, characterized in that: Determining the cause of the target abnormality according to the abnormal data and the abnormal log further includes: Taking the component corresponding to the abnormal data as the target component, and determining the abnormal log of the target abnormal level corresponding to the target component from the abnormal log; The cause of the target abnormality is determined according to the abnormality log of the target abnormality level corresponding to the target component and the abnormality data.
3. The monitoring method according to claim 2, characterized in that: The determining the cause of the target abnormality according to the abnormality log of the target abnormality level corresponding to the target component and the abnormality data further includes: Obtaining an exception code in an exception log of a target exception level corresponding to the target component; The abnormal code is matched with a preset abnormal code, and the abnormal causes corresponding to the successfully matched preset abnormal codes are combined and analyzed to determine the target abnormal cause.
4. The monitoring method according to claim 1, characterized in that: The performing aggregate analysis on the designated data after data processing to determine abnormal data in the designated data further includes: Perform variance calculation on the specified data of the same type after data processing to obtain variance values corresponding to multiple specified data of different types; The difference operation is performed on each type of designated data and its corresponding variance value, and the designated data with a difference greater than a preset difference value is regarded as abnormal data.
5. The monitoring method according to claim 1, characterized in that: The performing aggregate analysis on the designated data after data processing to determine abnormal data in the designated data further includes: Traverse the specified data after data processing, and use the traversed specified data as the target specified data; Acquire all target data on the target data processing chain corresponding to the target designated data, and match the value of each target data with the corresponding preset value to obtain a matching rate corresponding to the target data processing chain; If the matching rate is less than a preset matching rate, the target designated data is regarded as abnormal data.
6. The monitoring method according to any one of claims 1 to 5, characterized in that: The monitoring method further comprises: If the number of the target abnormal reasons reaches a preset number, or the current moment is within a preset alarm period, an alarm message is generated according to the target abnormal reason and the target abnormal log to be sent to different user terminals.
7. The monitoring method according to any one of claims 1 to 5, characterized in that: The monitoring method further comprises: Recording the initial abnormality log generated during the data processing process, and determining the abnormality log of the target abnormality level in the initial abnormality log; The abnormal logs of the target abnormal level are stored in a log database through a message queue, and the abnormal logs in the log database are displayed on the monitoring page.
8. A data anomaly monitoring device, characterized in that: The monitoring device comprises: An analysis and determination module is used to perform aggregate analysis on the specified data after data processing to determine abnormal data in the specified data; A cause determination module, used to determine the target abnormal cause according to the abnormal data and the abnormal log; wherein the abnormal log is a log recorded when the data processing is abnormal during the data processing process; A display module is used to display the target exception cause and the target exception log on a monitoring page; wherein the target exception log is a log in the exception log that is associated with the target exception cause.
9. An electronic device, characterized in that: include: Controller; A memory for storing one or more programs, which, when executed by the controller, enables the controller to implement the monitoring method described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a monitor of a computer, the computer is caused to execute the monitoring method according to any one of claims 1 to 7.
Citation Information
Cited By
Data display system and method based on big data stream processing
CN119760473A
