Abnormal data monitoring method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing and the field of financial science and technology, and discloses an abnormal data monitoring method, which comprises the following steps: capturing binlogs from a plurality of business databases, analyzing the binlogs to identify data change operation types, grouping the binlogs based on the operation types, and distributing grouping identifiers. And packaging the grouped binlogs into data packets, transmitting the data packets to different processing modules through message queues, screening to-be-detected data by the processing modules, analyzing whether abnormal conditions are met or not, marking the data meeting the abnormal conditions as abnormal data, generating alarm information and sending the alarm information to a service processing end. By capturing the binlog and performing grouping processing, a low-efficiency mode of independently developing a monitoring module for each business function in the prior art is avoided, and more efficient business system monitoring is realized. Meanwhile, data to be detected are screened and analyzed through the processing module, abnormal data are marked, and the real-time performance of abnormal monitoring and the stability of system operation are improved.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to an abnormal data monitoring method, device, equipment and storage medium. Background Art

[0002] In the financial sector, the normal operation of online business systems is crucial for capital circulation, customer transactions and business expansion. However, there is a risk of program anomalies in current online business systems, which, once occurring, may lead to serious financial losses and customer loss. This risk usually stems from bugs in the program or system configuration errors. If not discovered and handled in a timely manner, it will have a significant impact on the operation of the enterprise.

[0003] At present, in order to deal with such risks, the common practice is to develop a risk monitoring module for each business function. Although this method can monitor specific business modules, it is inefficient and has high development costs. Especially in large financial business systems, the complexity and interconnectivity of different functional modules increase the difficulty of developing and maintaining the monitoring system. In addition, the independent monitoring of multiple business modules is also prone to system performance degradation, because the monitoring logic is usually embedded in the business system, which affects the overall performance of the online system.

[0004] In addition to efficiency and performance issues, existing risk monitoring technologies also have shortcomings in terms of real-time performance. Many systems use timers to collect data for analysis on a regular basis, but this mechanism based on scheduled tasks often results in a delay in the discovery of anomalies. When an anomaly occurs in the system, it may have caused business interruption or loss, which in turn increases the difficulty of system recovery and loss control.

[0005] Existing risk monitoring methods require the development of independent monitoring modules for each business function, which increases development costs and reduces development efficiency. Since monitoring logic is usually embedded in business systems, existing risk monitoring methods will have a negative impact on system performance, especially in high-concurrency environments. At the same time, existing monitoring methods rely on timed data collection and analysis, resulting in anomaly detection not being real-time enough, affecting the timely handling of problems.

[0006] Therefore, the financial sector urgently needs a new solution that can efficiently, real-time and accurately monitor abnormal operations in business systems to avoid the risk of capital loss and user loss. Summary of the invention

[0007] The main purpose of the present invention is to provide an abnormal data monitoring method, device, equipment and storage medium, aiming to solve the technical problems in the prior art that business risk monitoring is inefficient and difficult to detect anomalies in real time, resulting in increased development costs and decreased system performance.

[0008] To achieve the above object, the present invention provides an abnormal data monitoring method, comprising:

[0009] Capturing binlog logs from multiple business databases, and parsing the binlog logs to identify data change operation types in the binlog logs;

[0010] Grouping the binlog logs based on the data change operation type, and assigning a group identifier to each group of binlog logs after grouping;

[0011] The grouped binlog logs are encapsulated into data packets and sent to the corresponding processing modules through the message queue. The data stream of each group of binlog logs corresponds to a different processing module.

[0012] Through each processing module, based on the configured preset standards, the data to be detected that meets the preset standards are filtered out from the corresponding binlog log;

[0013] Analyze whether the data to be detected meets the abnormal condition, and mark the data to be detected that meets the abnormal condition as abnormal data;

[0014] Generate alarm information based on the abnormal data, and send the alarm information to the business processing end.

[0015] Furthermore, to achieve the above object, the present invention provides an abnormal data monitoring device, comprising:

[0016] A log capture module, used to capture binlog logs from multiple business databases, and parse the binlog logs to identify the data change operation type in the binlog logs;

[0017] A log grouping module, used to group the binlog logs based on the data change operation type, and assign a group identifier to each group of binlog logs after grouping;

[0018] The data encapsulation and transmission module is used to encapsulate the grouped binlog logs into data packets and send the encapsulated binlog logs to the corresponding processing modules through the message queue. The data flow of each group of binlog logs corresponds to a different processing module;

[0019] The data screening module is used to screen out the data to be detected that meets the preset criteria from the corresponding binlog log based on the preset criteria configured by each processing module;

[0020] An anomaly detection module, used to analyze whether the data to be detected meets the abnormal condition, and mark the data to be detected that meets the abnormal condition as abnormal data;

[0021] The alarm generation and notification module is used to generate alarm information based on the abnormal data and send the alarm information to the business processing end.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an abnormal data monitoring program stored in the memory and executable on the processor, and the abnormal data monitoring program, when executed by the processor, implements the steps of the abnormal data monitoring method as described above.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an abnormal data monitoring program is stored, and when the abnormal data monitoring program is executed by a processor, the steps of the abnormal data monitoring method described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of data processing technology and financial technology, and discloses a method for monitoring abnormal data, which captures binlog logs from multiple business databases, parses binlog logs to identify data change operation types, groups binlog logs based on operation types, and assigns group identifiers. The grouped binlog logs are encapsulated into data packets, transmitted to different processing modules through message queues, and the processing modules screen the data to be detected, analyze whether the abnormal conditions are met, mark the data that meets the abnormal conditions as abnormal data, generate alarm information and send it to the business processing end. The present invention avoids the inefficient way of developing a monitoring module for each business function separately in the prior art by capturing binlog logs and performing group processing, and realizes more efficient business system monitoring. At the same time, the processing module screens and analyzes the data to be detected, and marks the abnormal data in time, which solves the problems of abnormal detection lag and system performance degradation, and improves the real-time nature of abnormal monitoring and the stability of system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of an abnormal data monitoring method in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a process flow of an abnormal data monitoring method according to an embodiment of the present invention;

[0028] Figure 3It is a functional module diagram of a preferred embodiment of the abnormal data monitoring device of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0030] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0032] The abnormal data monitoring method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user terminal communicates with the service terminal through a network. The service terminal can capture binlog logs from multiple business databases through the user terminal, parse the binlog logs to identify the data change operation type, group the binlog logs based on the operation type, and assign a group identifier. The grouped binlog logs are encapsulated as data packets, transmitted to different processing modules through message queues, the processing modules screen the data to be detected, analyze whether the abnormal conditions are met, mark the data that meets the abnormal conditions as abnormal data, generate alarm information and send it to the business processing terminal. The present invention avoids the inefficient way of developing a monitoring module for each business function separately in the prior art by capturing binlog logs and performing group processing, and realizes more efficient business system monitoring. At the same time, the processing module screens and analyzes the data to be detected, and marks the abnormal data in time, solves the problems of abnormal detection lag and system performance degradation, and improves the real-time nature of abnormal monitoring and the stability of system operation. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The service terminal can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of an abnormal data monitoring method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the abnormal data monitoring method proposed by the present invention includes the following steps:

[0035] S10, capturing binlog logs from multiple business databases, and parsing the binlog logs to identify data change operation types in the binlog logs;

[0036] In this embodiment, the binlog logging module is enabled in multiple business databases. This requires that the database supports the binlog logging function and can record all data change operations that occur in the database. The binlog log usually captures detailed records of operation types such as insert (INSERT), update (UPDATE), and delete (DELETE) for subsequent processing and analysis. By configuring the binlog logging function of the database system (such as MySQL's binlog_format is set to ROW mode), ensure that the change operation of each row of data is recorded. Tools such as Flink's binlog connector or Debezium can be used to capture binlog logs in multiple business databases in real time. These tools continuously obtain log change records in the database by connecting to the binlog interface of the database, ensuring that all change operations are recorded and transmitted to the next step for parsing.

[0037] After capturing the binlog log, you need to parse the content of each log record, especially to identify the operation type of the data change. These operation types usually include INSERT, UPDATE, and DELETE. The core of parsing binlog logs is to extract these operation types from the log structure for subsequent group processing and analysis. Use log parsing tools (such as Flinkbinlog connector or Debezium) to parse binlog logs. The parsing tool extracts the operation type information of each log by reading the metadata and record content in the log file. This information includes the type of operation (such as INSERT, UPDATE, DELETE), the table name and field involved, and the specific details of the data change (such as the old and new values ​​of the field). The parsed data change type is stored as structured information for subsequent operations.

[0038] Example: In the financial field, various business databases usually record a large number of transactions and fund changes. By capturing the binlog logs of these databases in real time and parsing the change operation type of each transaction (such as fund transfer in, transfer out, etc.), financial institutions can track each fund flow in real time. For example, when the system detects that a certain account has frequent fund transfer operations, it can mark it as a potential risk in time and transmit this information to the risk control module for further analysis and processing. This real-time capture and analysis mechanism can effectively reduce the risk of fund loss and improve the security and stability of financial services.

[0039] By capturing binlog logs from multiple business databases in real time and parsing the logs to identify the operation type of data changes, it is possible to track and process real-time data changes in business systems. This improves the real-time and accuracy of log data acquisition, ensures real-time monitoring of data changes, and reduces the lag in abnormal data detection.

[0040] S20, grouping the binlog logs based on the data change operation type, and assigning a group identifier to each group of binlog logs after grouping;

[0041] In this embodiment, after parsing the operation type (such as insert, update, delete) of the binlog log, the system needs to group the binlog log according to these operation types. This means that log entries with the same operation type are grouped together for subsequent processing. This grouping allows log data to be processed and analyzed independently according to the operation type, which helps to improve the efficiency of data processing. First, the system will classify the log data according to the parsed operation type. Common operation types include INSERT, UPDATE, and DELETE. The classification method can classify log entries of the same type into the same group by using a hash table or a mapping table. For example, all INSERT operation logs are grouped together, and all UPDATE operation logs are grouped together. This operation can be completed in memory for efficient group management.

[0042] After grouping, a unique group identifier needs to be assigned to each group of binlog logs. The group identifier is used to mark each group to ensure that the group log can be uniquely identified during subsequent processing, transmission, and tracking. The group identifier can be generated based on information such as the operation type, the number of logs, the data table structure, or the timestamp to ensure its uniqueness. The group identifier can be generated by generating a universally unique identifier (UUID) or based on a hash algorithm. When generating an identifier, you can combine elements such as the encoding of the operation type, timestamp information, and the number of log entries in the group. For example, the identifier of each group can contain the encoding of the operation type (such as INS for INSERT), the timestamp of the group (such as 20231001), and other identification information (such as the number of log entries). These identifiers will be attached to each group of logs to ensure that the groups are tracked and managed during message transmission or data processing.

[0043] Example: In the transaction system in the financial field, there are a large number of fund change operations on a daily basis, such as account recharge, fund withdrawal, transfer, etc. These operations will be recorded in the binlog logs of each database. By grouping the same type of fund operation logs (such as recharge logs, transfer logs) and assigning identifiers, financial institutions can process large amounts of fund transaction data more efficiently. For example, the system can handle all recharge operations independently and monitor whether there are abnormal behaviors, without mixing recharge operations with other fund operations. The identifier can also help the system track the processing progress of each group of fund operations, ensuring that all transaction data is correctly processed and stored, thereby improving the security and stability of fund transactions.

[0044] By grouping binlog logs according to the type of data change operation, the system can improve the efficiency of data processing, ensure that different types of operations are processed independently, and avoid data confusion. Assigning a group identifier to each group of logs ensures the traceability of log data in subsequent transmission and processing, reducing the risk of data loss and confusion.

[0045] S30, encapsulating the grouped binlog logs into data packets, and sending the encapsulated binlog logs to corresponding processing modules through message queues, wherein the data streams of each group of binlog logs correspond to different processing modules;

[0046] In the present embodiment, after completing binlog log grouping, each group of grouped logs needs to be encapsulated. The encapsulation process is to convert the grouped logs into data packets for easy transmission and processing over the network. The encapsulated data packets not only contain the content of the log, but also contain group identifiers and other necessary metadata information, such as verification information, the serial number of the log, etc., to ensure that the data is not damaged during transmission. The encapsulation process usually relies on a certain serialization format, such as JSON, Avro or Protobuf, which can package structured binlog log data into standardized binary or text data packets. The specific implementation can use a serialization library or tool to encapsulate the grouped logs and additional information together. During encapsulation, the content of the log (such as operation type, table name, field change) and the group identifier, verification information, etc. will be packaged into a complete data packet to ensure data integrity.

[0047] The encapsulated data packets need to be transmitted through the message queue. The message queue system is used to achieve reliable data transmission in distributed systems. Push the encapsulated log data packets to the message queue, and the message queue will route them to the corresponding processing module. The data stream of each group of binlog logs is transmitted to different processing modules for subsequent processing. Common message queue systems include Kafka, RabbitMQ or ActiveMQ. After the data packet is encapsulated, it is pushed as a message of the message queue. When sending, the system needs to configure the target processing module of the message queue (usually in the form of a topic or queue). For example, Kafka configures different topics for each different type of log grouping, and each topic corresponds to one or more processing modules to ensure that the grouped logs are sent to the corresponding modules. During the transmission process, the message queue will ensure the reliability of the data, such as providing a retry mechanism and data persistence function to prevent message loss.

[0048] Each group of encapsulated binlog logs will correspond to a specific processing module, and different grouped logs will be routed to different processing modules according to their operation type or other characteristics. Each processing module is responsible for processing operations such as analysis, screening, and anomaly detection on a specific type of log. The message queue will distribute data packets to different processing modules, which can be achieved by configuring different topics or queues. For example, Kafka will configure different topics for different binlog log types (such as INSERT, UPDATE, DELETE), and the processing module will pull log data from the message queue and process it according to the assigned topic. In this way, the system can process multiple log grouping data in parallel, improving the system's processing power and efficiency.

[0049] Example: In large-scale fund transaction systems in the financial field, fund flows generate a large number of database change logs. By grouping and encapsulating these logs into data packets, and transmitting them to different processing modules using Kafka, different fund operations can be independently monitored and processed. For example, all transfer operation logs can be sent to a dedicated transfer processing module, while recharge operation logs can be routed to another processing module. Through message queues, different types of fund operations can be processed in parallel and efficiently, reducing system load, and providing timely warnings of potential risks, thereby improving the overall stability and security of the system.

[0050] By encapsulating the grouped binlog logs into data packets and transmitting them to the processing module using message queues, efficient and reliable data distribution and processing are achieved. Different processing modules can process data streams from different groups in parallel, improving the system's concurrency and processing efficiency. At the same time, the use of message queues ensures the reliability of data transmission and reduces the risk of data loss.

[0051] S40, filtering out the data to be detected that meets the preset standard from the corresponding binlog log based on the preset standard configured by each processing module;

[0052] In this embodiment, each processing module screens the data to be detected that meets the requirements from the received binlog log data according to the preset standards defined in advance. The preset standards can be configured according to business needs, and usually include multiple dimensions such as data change operation type, field content, operation frequency, time window, etc. In the system, the processing module loads the preset standards configured by business experts or system administrators, which can be saved in configuration files or databases. When the system starts, these standards are dynamically loaded according to the configuration and applied to each processing module. The preset standards may include conditions such as "triggering detection when the amount of a single transaction exceeds a specific threshold" or "triggering detection when a specific field is modified". The processing module will screen the received binlog log data according to these standards to ensure that only data that matches the standards is processed.

[0053] The processing module filters the received binlog logs according to the preset standards and finds the data to be detected that meets the standards. The data to be detected refers to the data that meets the conditions in the log, which will further enter the anomaly detection or analysis stage. Each processing module reads the binlog log data from the message queue in real time and parses the content, such as operation type, data table, field change, operation time, etc. According to the preset standards, the processing module compares the field content and operation information of the log data one by one to filter out the records that meet the conditions. For example, if the preset standard requires filtering out all operations involving fund transfers, the processing module will match the operation records of the "transfer" type and use these qualified records as the data to be detected.

[0054] Example: In the financial system, a large number of fund transaction records are generated in the form of binlog logs. The processing module can filter these transaction logs according to the configured preset criteria. For example, a financial institution wants to monitor all single fund transfer operations exceeding 1 million. The processing module can set the preset criteria and filter out all qualified records based on the amount field. These qualified data to be tested will be marked as high-risk transactions and enter the next step of the abnormal analysis process. In this way, financial institutions can identify large or abnormal fund flows more quickly and ensure that business risks are handled in a timely manner.

[0055] Each processing module can flexibly respond to anomaly detection needs in different business scenarios by filtering the data to be detected based on the configured preset standards. The dynamic configuration and flexible adjustment of preset standards enable the system to filter out key data related to business logic in real time according to actual needs, avoiding the processing of irrelevant data and improving the operating efficiency of the system. At the same time, modular processing enables the system to maintain high efficiency and stability when processing multiple data in parallel.

[0056] S50, analyzing whether the data to be detected meets an abnormal condition, and marking the data to be detected that meets the abnormal condition as abnormal data;

[0057] In this embodiment, after filtering out the data to be detected that meet the preset standards, the processing module needs to further analyze whether these data meet the abnormal conditions. Abnormal conditions are pre-defined specific rules or thresholds for identifying potential abnormal operations. Abnormal conditions can be flexibly set according to business needs, usually involving transaction amounts, operation frequencies, field content changes, etc. The processing module first reads the filtered data to be detected, and then compares these data with the preset abnormal conditions. Abnormal conditions can be stored in a configuration file or a database, and the system will load these conditions and apply them to the analysis of the data to be detected during operation. For example, if the data to be detected involves fund operations, the abnormal conditions can be "a single transaction amount exceeds a specified threshold" or "multiple fund transfers in a short period of time". During analysis, the system will calculate or compare the various fields of the data to be detected (such as amount, number of operations, etc.), and determine whether they exceed the abnormal threshold. If the data to be detected meets the abnormal conditions, it will enter the next step for marking.

[0058] After completing the abnormal condition analysis, if the data to be detected meets these conditions, the system will mark it as abnormal data. The abnormal data mark can be a flag or label attached to the data to indicate that the data has potential risks or abnormal conditions and requires further processing or alarm. When the system detects that the data meets the abnormal conditions, it will add a flag to the data record or generate a corresponding abnormal identifier. For example, the processing module can add an "abnormal" field to the record that meets the abnormal conditions and mark it as high-risk data. This flag or identifier will be used in subsequent processing modules or alarm systems to take timely response measures. At the same time, the system will record the specific reasons for meeting the abnormal conditions (such as exceeding the amount threshold, frequent operations, etc.), and attach this information to the abnormal data for subsequent analysis and tracking.

[0059] Example description: In the financial system, anomaly detection is crucial to preventing financial risks. For example, frequent large-amount fund transfer operations in a short period of time in an account may trigger a system alarm. By analyzing the abnormal conditions of the data to be detected (such as each fund transfer record), the processing module can determine whether the account has exceeded the preset fund operation frequency or amount limit. Once it is detected that the abnormal conditions are met (such as "5 transfers were made to the same account within 10 minutes, with a total amount of more than 1 million yuan"), the system will mark these records as abnormal data, and generate an alarm in time to notify the relevant departments for further review. This mechanism plays an important role in preventing fraudulent transactions and financial risks.

[0060] By analyzing and marking abnormal conditions of the data to be detected, potential risks or abnormal operations in the business system can be effectively identified, and abnormal data that needs to be focused on can be marked in a timely manner. The ability to flexibly set and adjust abnormal conditions improves the system's adaptability to complex business scenarios and the accuracy of abnormal detection. The system's marking of abnormal data not only helps to issue timely alarms, but also provides clear records and basis for subsequent audits and data analysis.

[0061] S60, generating alarm information based on the abnormal data, and sending the alarm information to the business processing end.

[0062] In this embodiment, after the system marks the data that meets the abnormal conditions, it is necessary to generate alarm information based on these abnormal data. The generation of alarm information is based on the content of the abnormal data and the relevant abnormal characteristics, usually including the details of the abnormal data (such as transaction amount, operation type, time, etc.), the conditions that trigger the abnormality, and the impact that the abnormality may have on the business. After identifying the abnormal data, the processing module first extracts key information from the abnormal data, such as operation time, operation type, field content changes, etc. Then, the system organizes this information into an alarm message according to the preset alarm template. The alarm information can be structured data (such as JSON or XML format) for easy transmission between different systems. The alarm information can also be classified according to the severity of the abnormality (such as high, medium, and low levels) so that the downstream processing module or the business end system can take corresponding measures in time.

[0063] After the alarm information is generated, it needs to be sent to the business processing end in a timely manner, usually the monitoring system or the business team responsible for handling exceptions. There are many ways to send alarm information, including notification through message queues, HTTP API, or through email, SMS, etc. The generated alarm information is sent to the business processing end through a messaging mechanism. Usually, a message queue (such as Kafka, RabbitMQ) or a push API (such as RESTful API) can be used for real-time notification. After the processing end receives the alarm information, different business processes or response mechanisms can be triggered according to the priority and content of the alarm. For high-priority alarms, the system can automatically trigger the predetermined emergency handling procedures, while low-priority alarms can be recorded in the monitoring log for manual follow-up processing. Each alarm sending operation will record the sending status to ensure the successful delivery and tracking of the alarm information.

[0064] Example: In the financial sector, timely processing of abnormal data is crucial to preventing fraud and financial losses. Assuming that a bank system detects that multiple accounts have undergone high-frequency and large-amount fund transfer operations in a short period of time, the processing module will mark these fund flow data as abnormal and generate an alarm message. The alarm message will contain a detailed description of the abnormality, such as account number, operation time, transfer amount, and the abnormal condition that triggered it. The alarm information is sent to the bank's monitoring system in real time through the Kafka message queue, and the monitoring system will immediately notify the risk management team for processing. High-priority alarms may also trigger automatic freezing of accounts or suspension of fund transfers to ensure that risks are controlled at the first time and possible financial losses are avoided.

[0065] By generating alarm information based on abnormal data and sending it to the business processing end, real-time monitoring and response to the business system can be achieved, ensuring that potential business risks are discovered and handled in a timely manner. The system's response speed and accuracy are improved, and the lag time for abnormal processing is reduced. At the same time, the classification and priority setting of alarm information further enhances the system's emergency response capabilities, allowing businesses to obtain feedback on abnormal situations in the shortest possible time.

[0066] The present invention relates to the fields of data processing technology and financial technology, and discloses a method for monitoring abnormal data, which captures binlog logs from multiple business databases, parses binlog logs to identify data change operation types, groups binlog logs based on operation types, and assigns group identifiers. The grouped binlog logs are encapsulated as data packets, transmitted to different processing modules through message queues, and the processing modules screen the data to be detected, analyze whether the abnormal conditions are met, mark the data that meets the abnormal conditions as abnormal data, generate alarm information and send it to the business processing end. The present invention avoids the inefficient way of developing a monitoring module for each business function separately in the prior art by capturing binlog logs and performing group processing, and realizes more efficient business system monitoring. At the same time, the processing module screens and analyzes the data to be detected, and marks the abnormal data in time, which solves the problems of abnormal detection lag and system performance degradation, and improves the real-time and stability of abnormal monitoring and system operation.

[0067] In one embodiment, in the above S10, capturing binlog logs from multiple business databases includes:

[0068] S101, enable the binlog logging module in each business database and record all data change operations, including insert, update or delete;

[0069] S102, setting a corresponding log capture frequency according to the frequency of data change operations and the data change amount threshold in each business database;

[0070] S103: Capture binlog logs from a corresponding business database based on the log capture frequency.

[0071] In this embodiment, each business database needs to enable the binlog logging function to ensure that each data change operation is recorded. Data change operations usually include insert (INSERT), update (UPDATE) or delete (DELETE), etc. The logging function can track each database operation and provide a data source for subsequent analysis and monitoring. Enable the binlog logging function in the database. Taking MySQL as an example, you can set binlog_format to ROW mode to record the change details of each data row. In addition, the administrator needs to ensure that the logging function of the database is correctly enabled and configure the storage location, size limit and retention policy of the log according to business needs. The logging module will automatically capture every insert, update or delete operation involving table data and write it to the binlog log.

[0072] The frequency and amount of data changes in different business databases vary, so the system needs to set a corresponding log capture frequency for each database. The capture frequency determines how quickly the system can obtain the latest data in the binlog log. The frequency can be dynamically configured based on the operation frequency of the database, the amount of data changes, or business needs. The system sets an appropriate log capture frequency for each database by monitoring the operation frequency (such as the number of operations per second TPS) and the amount of data changes (such as the amount of data modified per day) of each business database. For example, a database with frequent data modifications can set a higher capture frequency (such as once per minute), while a database with less data changes can set a lower capture frequency (such as once per hour). This setting can be adjusted in real time through the database monitoring system to ensure that the capture frequency is consistent with the dynamic changes in business data operations.

[0073] After setting the log capture frequency, the system captures binlog logs in real time from each business database according to this frequency. The capture process is to obtain the latest binlog logs from the database to ensure that the system can process and monitor data change operations in real time. The system will periodically extract binlog logs from the business database according to the configured capture frequency. Real-time stream processing tools (such as Flink or Debezium) can be used to obtain binlog log data from the database and pass it to subsequent data processing modules. During the capture process, the system will record the log offset of each capture to ensure that the next capture can continue from where the last capture ended without missing any change operations. At the same time, the system needs to monitor the capture delay time to ensure that the logs can be transmitted to the processing module in a timely manner.

[0074] Example description: In the financial field, business systems may record a large amount of fund transaction data and account change operations. By enabling the binlog logging function for each business database, financial institutions can track all fund change operations in detail. For trading systems with high activity, the system will set a higher log capture frequency to ensure that each fund transfer, recharge or withdrawal operation can be captured and processed within a few minutes. For relatively static databases, such as databases that store historical account information, a lower capture frequency can be set. By dynamically adjusting the log capture frequency, financial institutions can ensure that any abnormal fund operations are detected in the shortest time, while avoiding unnecessary system overhead and improving the overall performance of the system.

[0075] This embodiment can efficiently capture the binlog logs of each database by dynamically setting the log capture frequency according to the operation frequency and data change amount of the business database, avoiding resource waste and capture delay. At the same time, based on the set frequency, the binlog logs are captured in real time to ensure that the system can obtain the latest database change operation data in a timely manner, thereby improving the real-time and accuracy of monitoring.

[0076] In one embodiment, the above S20 includes:

[0077] S201, preliminarily classify the binlog logs according to the data change operation types in the binlog logs, and generate preliminary binlog log classification results;

[0078] S202, based on the preliminary binlog classification result, grouping binlog logs of the same type according to the data table structure to generate grouped binlog logs;

[0079] S203, generating a group identifier according to the number of logs in each group of binlog logs, the data table structure and the timestamp, and assigning the group identifier to each corresponding group of binlog logs;

[0080] S204, storing the group identifier in a memory for tracking the source of each group of binlog logs.

[0081] In this embodiment, the system will preliminarily classify the logs according to the data change operation type (such as insert, update or delete) in the binlog log. This means that the system will parse each binlog log and extract the operation type therein so as to perform more detailed grouping processing later. The preliminary classification results can provide a basis for subsequent log grouping. The system extracts the operation type field from the binlog log and preliminarily classifies the logs according to the operation type. For example, all logs of INSERT operations will be classified into one category, and logs of UPDATE operations will be classified into another category. This process can be implemented through string matching or hash mapping. The classified results will generate a preliminary classification record for subsequent log grouping operations.

[0082] After completing the preliminary classification, the system further groups binlog logs of the same type according to the data table structure involved in the logs. The purpose of this step is to group logs for the same data table under the same operation type into one group. The data table structure includes information such as the table name and the field definition of the table, which determines how to group the logs in a more fine-grained manner. The system further groups the classified binlog logs by parsing the table name, field and other information in the log. For example, for the INSERT operation log, the system will group all logs involving the same data table into one subgroup. The logic of grouping can be implemented by hashing or mapping the table name and field structure. Each group will save the corresponding log entries and the data table structure information to which they belong.

[0083] After completing the log grouping, the system needs to generate a unique group identifier for each group. The generation of the group identifier is based on the number of logs, the data table structure, and the timestamp to ensure that each group can be uniquely identified. The group identifier will be assigned to the corresponding binlog log group for subsequent processing and tracking. The generation of the group identifier can use multiple parameter combinations, such as the number of log entries, the name and structure of the data table, and timestamp information. By combining these parameters for hashing or generating a UUID, a unique identifier can be created. The generated identifier will be attached to each group to ensure that each group can be uniquely identified and managed in subsequent processing steps.

[0084] The system stores the identifier generated for each group in memory so that the source of each group of binlog logs can be tracked later. By storing the group identifier, the system can effectively track the flow and processing status of log groups at different stages of log processing to ensure data traceability. The generated group identifier will be stored in memory, for example, using a hash table or mapping structure. When storing the identifier, the system will record the relevant metadata of the group log so that the source of each group of logs can be quickly queried and tracked during subsequent log processing, transmission, and analysis. The group identifier in memory can be called by the log processing module to ensure the consistency and integrity of the log at different processing stages.

[0085] Example: In the financial system, a large number of fund operations such as transfers, top-ups, and withdrawals are recorded in the binlog log of the database. By grouping these operations by type and generating a unique identifier for each group of logs, the system can monitor and process each type of fund operation independently. For example, all operation logs involving customer fund transfers are grouped together and a specific identifier is generated. This identifier tracks all processing stages of the fund transfer operation, ensuring that the logs can be correctly circulated and stored to prevent potential abnormal operations or data loss.

[0086] The present embodiment improves the management efficiency and processing accuracy of log data by grouping binlog logs based on the data change operation type and assigning a unique identifier to each group of logs. Log grouping enables the system to process and analyze data change operations more systematically, and the generation and storage of group identifiers ensure the traceability of log data and avoid the risk of data loss or processing confusion. Compared with traditional disordered log processing methods, the introduction of grouping and identifiers significantly improves the processing performance and data consistency of the system.

[0087] In one embodiment, in the above S30, the grouped binlog logs are encapsulated into data packets, and the encapsulated binlog logs are sent to corresponding processing modules through a message queue, including:

[0088] S301, determining a data encapsulation format according to the parsing requirements of the processing module and the structural complexity of the binlog log, and encapsulating the binlog log based on the data encapsulation format;

[0089] S302, during the encapsulation process, the packet identifier and verification information are added to the binlog log data packet;

[0090] S303, transmitting each group of encapsulated binlog log data packets to the corresponding processing module through the message queue, and recording the transmission status of each group of binlog log data packets;

[0091] S304, monitoring the transmission status of the message queue, and if a transmission failure or delay is detected, executing a retry mechanism or triggering an error alarm.

[0092] In this embodiment, the system selects a suitable data encapsulation format according to the specific requirements of the processing module and the complexity of the binlog log. Different processing modules may require different encapsulation formats to ensure that the data can be correctly parsed and processed. Common data encapsulation formats include JSON, Avro, Protobuf, etc., which have different structural and serialization performances and are suitable for different scenarios. The system selects a suitable data encapsulation format according to the parsing requirements of the processing module (such as requirements for field order and data type). For logs with simple structures, JSON format can be used for encapsulation, which is easy to parse and transmit; for complex logs that require efficient serialization and parsing, binary formats such as Avro or Protobuf can be selected for encapsulation. During the encapsulation process, the metadata information such as the field content, operation type and timestamp of the log will be encoded and packaged according to the selected encapsulation format to form a standardized data packet.

[0093] During the encapsulation process, the system needs to add the previously generated group identifier and verification information to each data packet. The group identifier ensures the uniqueness of the data packet, while the verification information is used to verify whether the data has been tampered with or damaged during transmission, ensuring the integrity and security of the data. When encapsulating binlog logs, the system adds the group identifier to the data packet as an additional metadata field. The identifier ensures that the data packet can be distinguished from other group logs. The verification information is generated by performing a hash operation (such as SHA-256) on the log content and is used for integrity verification after the transmission is completed. The data packet contains both the content of the binlog log and the group identifier and verification information, thereby ensuring the uniqueness and reliability of the data packet.

[0094] After encapsulation, the system needs to send the data packet to the corresponding processing module through the message queue. Message queues are used to achieve reliable data transmission and decoupling in distributed systems. The transmission status of each data packet needs to be recorded for subsequent tracking or processing failures. The system uses message queues (such as Kafka, RabbitMQ) as a data transmission mechanism to send the encapsulated binlog log data packets to the corresponding processing modules in sequence. Each group of data packets will be transmitted to a specific processing module through the configured queue or topic, and the processing module will obtain the log data packet according to the topic name in the message queue. During the transmission process, the system will record the transmission status of each data packet, including information such as transmission success, failure, or number of retries, for subsequent query and processing.

[0095] In order to ensure that the data packet can be successfully transmitted, the system needs to monitor the status of the message queue in real time. If a transmission delay or failure is found, the system will trigger a retry mechanism to resend the data packet, or issue an error alarm to notify the administrator to handle it. The system will continuously monitor the data transmission status in the message queue to check whether there is any transmission delay, loss or failure. If a transmission failure is detected, the system will automatically execute the retry mechanism and try to retransmit the data packet. The number of retries and the time interval can be configured according to business needs. If multiple retries fail, the system will trigger an error alarm and send an alarm message to the operation and maintenance personnel to remind them to manually handle the problem. Message queue monitoring tools such as Kafka Manager or Prometheus can be used to achieve real-time monitoring of transmission status.

[0096] This embodiment ensures efficient transmission and reliable processing of log data by encapsulating the grouped binlog logs into data packets and transmitting them to the processing module through message queues. The flexible choice of encapsulation format enables the system to adapt to different processing requirements, and the addition of packet identifiers and verification information enhances the traceability and integrity of the data. At the same time, through the use of message queues, the system realizes the decoupling and expansion of distributed processing modules, and improves the security and stability of data transmission by real-time monitoring of transmission status, reducing the risk of data loss.

[0097] In one embodiment, the above S40 includes:

[0098] S401, configuring a preset standard for each processing module, wherein the preset standard includes a data operation type standard, a field content standard, an operation frequency standard, or an operation time window standard;

[0099] S402, when receiving a binlog log, the processing module parses the binlog log and selects the data to be detected that meets the preset standard from the binlog log.

[0100] In the present embodiment, each processing module sets preset standards according to its specific monitoring and detection requirements. These preset standards can include conditions of different dimensions, such as data operation type (insert, update, delete, etc.), field content (such as changes in field values), operation frequency (number of operations per unit time) and operation time window (data operations within a specific time period). The system administrator or monitoring module will configure these preset standards for each processing module. These standards can be defined through configuration files, management interfaces or databases. The operation type standard is used to filter specific types of database operations (for example, only monitor INSERT operations). The field content standard is used to set the value of certain fields to trigger the screening (for example, a specific field value is greater than a certain threshold). The operation frequency standard can monitor the abnormal operation frequency within a certain period of time, and the operation time window standard is used to analyze the data changes in a specific time period. All these standards will be loaded into the processing module for log screening.

[0101] When the processing module receives the binlog log transmitted from the message queue, the module will parse the log content according to the preset standards and filter out the data to be detected that meets these standards. This process involves decoding the binlog log, extracting fields, and matching conditions. The processing module receives the binlog log from the message queue in real time and parses the log. During the parsing process, the module extracts key field information in the log, such as operation type, table name, field content, operation time, etc. Then, the module compares these fields with the configured preset standards to determine whether the log data meets the conditions. For example, if the field value in a log exceeds the preset threshold, or the operation type is INSERT and occurs within the preset time window, the system will mark the log as data to be detected for subsequent abnormal analysis.

[0102] This embodiment provides a flexible and efficient abnormal data screening mechanism by configuring preset standards for each processing module and screening the data to be detected according to these standards. The flexible configuration of preset standards allows the system to customize monitoring conditions according to different business needs, improving the accuracy of abnormal detection. The processing module screens log data in real time based on these standards, ensuring efficient log processing and rapid identification of potential risks.

[0103] In one embodiment, the above S50 includes:

[0104] S501, based on historical abnormal records, setting abnormal conditions for detecting abnormal data for each processing module, wherein the abnormal conditions include an abnormal threshold range, an abnormal data change rate, an abnormal data pattern, or an abnormal operation time window;

[0105] S502, analyzing the data to be detected to determine whether the data to be detected meets the abnormal condition;

[0106] S503, if the data to be detected meets the abnormal condition, marking the data to be detected as abnormal data, and adding an abnormal identifier to the abnormal data;

[0107] S504, recording abnormal information of the abnormal data, where the abnormal information includes the content, source, detection time and abnormal type of the abnormal data.

[0108] In this embodiment, the setting of abnormal conditions is based on the abnormal data records that have occurred in the history of the system. The purpose is to summarize common abnormal patterns and set appropriate conditions to automatically detect new abnormal situations. Abnormal conditions can include multiple dimensions, such as abnormal threshold range, data change rate, data mode and operation time window, etc., which are specifically derived from historical data analysis. The system will analyze and summarize past abnormal events and extract common abnormal features (such as the amount range of fund operations, abnormal operation frequency, etc.). The system administrator or algorithm module sets the conditions for detecting abnormal data for each processing module based on these historical features. The abnormal threshold range can refer to the set upper and lower limits (such as the upper and lower limits of the amount), the abnormal data change rate refers to the rate at which the data changes sharply in a short period of time, the abnormal data pattern can be a sudden change in the operation mode, and the operation time window can be used to monitor operations that occur during abnormal working hours.

[0109] After obtaining the data to be detected, the processing module will compare and analyze these data with the pre-set abnormal conditions. Through multi-dimensional analysis of the data to be detected, it is determined whether there are situations that exceed the threshold range, abnormal data change rate or other pattern changes. If the data to be detected meets these abnormal conditions, the system will identify it as a potential anomaly. The processing module will parse the received data to be detected in real time, and compare each data with the preset abnormal threshold, change rate, data pattern and other standards. For example, whether the amount of a certain fund operation exceeds the preset threshold range (such as more than 1 million yuan), or whether the operation frequency of a certain user is abnormal (such as multiple operations within one minute). These comparison operations can be performed through a rule engine or a machine learning model to ensure accurate identification of potential anomalies.

[0110] When the data to be detected is identified as abnormal data, the system will mark the data and add an abnormal flag to ensure that the data is given priority attention and processing in the subsequent processing flow. The abnormal flag is a special label for the data, indicating that the data needs further analysis or action. The system adds an "abnormal" flag to each piece of data to be detected that meets the abnormal conditions, usually by setting a flag bit or field. For example, the system can add an "is_anomaly" field to the abnormal data with a value of "true" to indicate that the data is abnormal data. The additional abnormal flag may also contain the specific conditions that triggered the abnormal data, such as "exceeding the amount threshold" or "high-frequency operation". This flag will be used in subsequent alarms or processing mechanisms to ensure that the system can respond to abnormal data in a timely manner.

[0111] The system needs to record the relevant information of each abnormal data in detail for subsequent tracking and analysis. The abnormal information includes the specific content of the abnormal data (such as the operation type and field value), the source of the data (such as the user or system module of the operation), the detection time and the abnormal type (such as the threshold exceeded or the operation frequency is abnormal). The processing module will record all relevant information of the abnormal data in the log or database. The storage format of the abnormal information can be structured, such as JSON or database table records. The recorded information can include the specific content of the abnormal data (such as the table, field, value of the operation), the conditions that trigger the abnormality, the timestamp of the detection, and the classification of the abnormality (such as abnormal amount, abnormal frequency, etc.). These records will be used for subsequent business processing, auditing or reporting.

[0112] This embodiment realizes automatic marking and processing of abnormal data by analyzing and comparing the data to be detected with the preset abnormal conditions. The system can dynamically set reasonable abnormal conditions based on historical abnormal records, improving the accuracy of abnormal data identification. At the same time, the system adds detailed abnormal identification and records to each abnormal data to facilitate subsequent analysis and tracking.

[0113] In one embodiment, the above S60 includes:

[0114] S601, based on the abnormal information of the abnormal data, classify the alarm levels, wherein the alarm levels include high priority alarms and low priority alarms;

[0115] S602, generating corresponding alarm information according to the alarm level and the abnormal data, the alarm information including the content, source, abnormality type, detection time and business impact analysis of the abnormal data;

[0116] S603, if the alarm level is a high priority alarm, send the alarm information to the service processing terminal;

[0117] S604: If the alarm level is a low priority alarm, the alarm information is recorded in a low priority alarm log, and the low priority alarm log is sent to the service processing end after each preset time interval.

[0118] In this embodiment, before generating the alarm information, the system first divides the alarm level according to the severity of the abnormal data. Usually, the alarm level is divided into high priority and low priority, so that the system can perform different processing according to different alarm levels. High priority alarms mean that the system needs to take immediate measures, while low priority alarms may not need to be processed immediately. The processing module will make judgments based on the information in the abnormal data (such as the severity of the operation, the scope of impact, etc.) and divide the alarm level. For example, the system determines whether the alarm belongs to high priority based on the business impact of the data (such as large-scale fund operations) and abnormal triggering conditions (such as high-frequency trading, etc.). If the abnormal data involves major business risks (such as abnormal transfer of a large amount of funds), the system will mark it as a high priority alarm; while smaller risk events may be classified as low priority alarms.

[0119] Once the alarm level is determined, the system will generate corresponding alarm information according to different levels. The alarm information includes the content, source, type of anomaly, detection time, etc. of the abnormal data, and also needs to include business impact analysis so that the business processing end can understand the potential impact of the anomaly. The processing module organizes the detailed information of the abnormal data and generates structured alarm information. The alarm information will include specific fields of abnormal data (such as the account operated, transaction amount, time, etc.), source information (IP address of the operation source, user ID, etc.), detection time, triggered abnormal conditions, and impact assessment on the business (such as possible financial risks). According to different alarm levels, the system can generate more detailed high-priority alarm information or simplified low-priority alarm information. The generation format of the alarm information can be JSON, XML, etc., which is convenient for transmission and processing.

[0120] When an alarm generated by the system is marked as high priority, the system needs to send the alarm information to the business processing end immediately. This type of alarm information usually involves urgent or high-risk business events. The business processing end needs to respond quickly and take action after receiving the high-priority alarm. The system will send high-priority alarm information to the business processing end through a messaging mechanism (such as a message queue or API push). For high-priority alarms, the system can also set additional notification methods, such as reminding relevant processing personnel through SMS, email or instant messaging. The transmission of high-priority alarms needs to be fast and reliable to ensure that the business processing end can respond in a timely manner.

[0121] For low-priority alarms, the system does not need to send them to the business processing end immediately. Instead, the system records these alarm information in the log and sends them in batches at set time intervals. This can effectively reduce the load on the system while ensuring that low-priority alarms are not ignored. When the system identifies a low-priority alarm, it records the alarm information in a low-priority alarm log file or database. The system sends these low-priority alarms to the business processing end in batches according to the configured time interval (such as every hour or every day). The transmission method can be the same as that of high-priority alarms, but it does not need to be real-time and urgent. In addition, the system can regularly check and send low-priority alarm information through a scheduled task.

[0122] This embodiment improves the efficiency of abnormal data processing and emergency response capabilities by dividing alarm levels and generating and transmitting alarm information according to the levels. High-priority alarms can notify the business processing end in the shortest time to ensure that major risks can be handled in a timely manner; while low-priority alarms can reduce system load through batch processing to avoid frequent alarms interrupting business processes.

[0123] In one embodiment, an abnormal data monitoring device is provided, and the abnormal data monitoring device corresponds one-to-one to the abnormal data monitoring method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the abnormal data monitoring device of the present invention. Log capture module 10, log grouping module 20, data encapsulation and transmission module 30, data screening module 40, abnormality detection module 50 and alarm generation and notification module 60. The functional modules are described in detail as follows:

[0124] The log capture module 10 is used to capture binlog logs from multiple business databases and parse the binlog logs to identify the data change operation type in the binlog logs;

[0125] A log grouping module 20, configured to group the binlog logs based on the data change operation type and assign a grouping identifier to each group of binlog logs;

[0126] The data encapsulation and transmission module 30 is used to encapsulate the grouped binlog logs into data packets and send the encapsulated binlog logs to the corresponding processing modules through the message queue. The data flow of each group of binlog logs corresponds to a different processing module.

[0127] The data screening module 40 is used to screen out the data to be detected that meets the preset standard from the corresponding binlog log based on the preset standard configured by each processing module;

[0128] An abnormality detection module 50, used to analyze whether the data to be detected meets the abnormality condition, and mark the data to be detected that meets the abnormality condition as abnormal data;

[0129] The alarm generation and notification module 60 is used to generate alarm information based on the abnormal data and send the alarm information to the business processing end.

[0130] In one embodiment, the log capture module 10 is specifically used to:

[0131] Enable the binlog logging module in each business database and record all data change operations, including insert, update, or delete;

[0132] Set the corresponding log capture frequency based on the frequency of data change operations and data change threshold in each business database;

[0133] Capture binlog logs from the corresponding business database based on the log capture frequency.

[0134] In one embodiment, the log grouping module 20 is specifically used to:

[0135] Preliminarily classify binlog logs according to the data change operation types in the binlog logs, and generate preliminary binlog log classification results;

[0136] Based on the preliminary binlog classification results, binlog logs of the same type are grouped according to the data table structure to generate grouped binlog logs;

[0137] Generate a group identifier based on the number of logs in each group of binlog logs, the data table structure and the timestamp, and assign the group identifier to each corresponding group of binlog logs;

[0138] The group identifier is stored in memory to track the source of each group of binlog logs.

[0139] In one embodiment, the data encapsulation and transmission module 30 is specifically used for:

[0140] Determine the data encapsulation format according to the parsing requirements of the processing module and the structural complexity of the binlog log, and encapsulate the binlog log based on the data encapsulation format;

[0141] During the encapsulation process, the group identifier and verification information are added to the binlog log data packet;

[0142] Transmit each group of encapsulated binlog log data packets to the corresponding processing module through the message queue, and record the transmission status of each group of binlog log data packets;

[0143] Monitor the transmission status of the message queue. If a transmission failure or delay is detected, a retry mechanism is executed or an error alarm is triggered.

[0144] In one embodiment, the data screening module 40 is specifically used to:

[0145] Configuring preset standards for each processing module, the preset standards including data operation type standards, field content standards, operation frequency standards or operation time window standards;

[0146] When receiving the binlog log, the processing module parses the binlog log and filters out the data to be detected that meets the preset standard from the binlog log.

[0147] In one embodiment, the anomaly detection module 50 is specifically configured to:

[0148] Based on historical abnormal records, setting abnormal conditions for detecting abnormal data for each processing module, wherein the abnormal conditions include an abnormal threshold range, an abnormal data change rate, an abnormal data pattern, or an abnormal operation time window;

[0149] Analyze the data to be detected to determine whether the data to be detected meets the abnormal condition;

[0150] If the data to be detected meets the abnormal condition, the data to be detected is marked as abnormal data, and an abnormal identifier is added to the abnormal data;

[0151] The abnormal information of the abnormal data is recorded, wherein the abnormal information includes the content, source, detection time and abnormal type of the abnormal data.

[0152] In one embodiment, the alarm generation and notification module 60 is specifically used to:

[0153] Based on the abnormal information of the abnormal data, the alarm level is divided, and the alarm level includes a high priority alarm and a low priority alarm;

[0154] Generate corresponding alarm information according to the alarm level and the abnormal data, the alarm information including the content, source, abnormal type, detection time and business impact analysis of the abnormal data;

[0155] If the alarm level is a high priority alarm, the alarm information is sent to the service processing terminal;

[0156] If the alarm level is a low priority alarm, the alarm information is recorded in a low priority alarm log, and the low priority alarm log is sent to the service processing end after each preset time interval.

[0157] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the service side of an abnormal data monitoring method are implemented.

[0158] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user-side of an abnormal data monitoring method

[0159] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0160] Capturing binlog logs from multiple business databases, and parsing the binlog logs to identify data change operation types in the binlog logs;

[0161] Grouping the binlog logs based on the data change operation type, and assigning a group identifier to each group of binlog logs after grouping;

[0162] The grouped binlog logs are encapsulated into data packets and sent to the corresponding processing modules through the message queue. The data stream of each group of binlog logs corresponds to a different processing module.

[0163] Through each processing module, based on the configured preset standards, the data to be detected that meets the preset standards are filtered out from the corresponding binlog log;

[0164] Analyze whether the data to be detected meets the abnormal condition, and mark the data to be detected that meets the abnormal condition as abnormal data;

[0165] Generate alarm information based on the abnormal data, and send the alarm information to the business processing end.

[0166] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0167] Capturing binlog logs from multiple business databases, and parsing the binlog logs to identify data change operation types in the binlog logs;

[0168] Grouping the binlog logs based on the data change operation type, and assigning a group identifier to each group of binlog logs after grouping;

[0169] The grouped binlog logs are encapsulated into data packets and sent to the corresponding processing modules through the message queue. The data stream of each group of binlog logs corresponds to a different processing module.

[0170] Through each processing module, based on the configured preset standards, the data to be detected that meets the preset standards are filtered out from the corresponding binlog log;

[0171] Analyze whether the data to be detected meets the abnormal condition, and mark the data to be detected that meets the abnormal condition as abnormal data;

[0172] Generate alarm information based on the abnormal data, and send the alarm information to the business processing end.

[0173] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0175] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0176] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for monitoring abnormal data, characterized in that: The following steps are involved: Capturing binlog logs from multiple business databases, and parsing the binlog logs to identify data change operation types in the binlog logs; Grouping the binlog logs based on the data change operation type, and assigning a group identifier to each group of binlog logs after grouping; The grouped binlog logs are encapsulated into data packets and sent to the corresponding processing modules through the message queue. The data stream of each group of binlog logs corresponds to a different processing module. Through each processing module, based on the configured preset standards, the data to be detected that meets the preset standards are filtered out from the corresponding binlog log; Analyze whether the data to be detected meets the abnormal condition, and mark the data to be detected that meets the abnormal condition as abnormal data; Generate alarm information based on the abnormal data, and send the alarm information to the business processing end.

2. The abnormal data monitoring method according to claim 1, characterized in that: Capture binlog logs from multiple business databases, including: Enable the binlog logging module in each business database and record all data change operations, including insert, update, or delete; Set the corresponding log capture frequency based on the frequency of data change operations and data change threshold in each business database; Capture binlog logs from the corresponding business database based on the log capture frequency.

3. The abnormal data monitoring method according to claim 1, characterized in that: The binlog logs are grouped based on the data change operation type, and a group identifier is assigned to each group of binlog logs after the grouping, including: Preliminarily classify binlog logs according to the data change operation types in the binlog logs, and generate preliminary binlog log classification results; Based on the preliminary binlog classification results, binlog logs of the same type are grouped according to the data table structure to generate grouped binlog logs; Generate a group identifier based on the number of logs in each group of binlog logs, the data table structure and the timestamp, and assign the group identifier to each corresponding group of binlog logs; The group identifier is stored in memory to track the source of each group of binlog logs.

4. The abnormal data monitoring method according to claim 1, characterized in that: Encapsulate the grouped binlog logs into data packets and send the encapsulated binlog logs to the corresponding processing modules through the message queue, including: Determine the data encapsulation format according to the parsing requirements of the processing module and the structural complexity of the binlog log, and encapsulate the binlog log based on the data encapsulation format; During the encapsulation process, the group identifier and verification information are added to the binlog log data packet; Transmit each group of encapsulated binlog log data packets to the corresponding processing module through the message queue, and record the transmission status of each group of binlog log data packets; Monitor the transmission status of the message queue. If a transmission failure or delay is detected, a retry mechanism is executed or an error alarm is triggered.

5. The abnormal data monitoring method as claimed in claim 1, characterized in that: Each processing module selects the data to be detected that meets the preset standard from the corresponding binlog log based on the preset standard configured, including: Configuring preset standards for each processing module, the preset standards include data operation type standards, field content standards, operation frequency standards or operation time window standards; When receiving the binlog log, the processing module parses the binlog log and filters out the data to be detected that meets the preset standard from the binlog log.

6. The abnormal data monitoring method according to claim 1, characterized in that: Analyzing whether the data to be detected meets an abnormal condition, and marking the data to be detected that meets the abnormal condition as abnormal data, includes: Based on historical abnormal records, setting abnormal conditions for detecting abnormal data for each processing module, wherein the abnormal conditions include an abnormal threshold range, an abnormal data change rate, an abnormal data pattern, or an abnormal operation time window; Analyze the data to be detected to determine whether the data to be detected meets the abnormal condition; If the data to be detected meets the abnormal condition, the data to be detected is marked as abnormal data, and an abnormal identifier is added to the abnormal data; The abnormal information of the abnormal data is recorded, wherein the abnormal information includes the content, source, detection time and abnormal type of the abnormal data.

7. The abnormal data monitoring method according to claim 1, characterized in that: Generating alarm information based on the abnormal data and sending the alarm information to the business processing end includes: Based on the abnormal information of the abnormal data, the alarm level is divided, and the alarm level includes a high priority alarm and a low priority alarm; Generate corresponding alarm information according to the alarm level and the abnormal data, the alarm information including the content, source, abnormal type, detection time and business impact analysis of the abnormal data; If the alarm level is a high priority alarm, the alarm information is sent to the service processing terminal; If the alarm level is a low priority alarm, the alarm information is recorded in a low priority alarm log, and the low priority alarm log is sent to the service processing end after each preset time interval.

8. An abnormal data monitoring device, characterized in that: The abnormal data monitoring device comprises: A log capture module, used to capture binlog logs from multiple business databases, and parse the binlog logs to identify the data change operation type in the binlog logs; A log grouping module, used to group the binlog logs based on the data change operation type, and assign a group identifier to each group of binlog logs after grouping; The data encapsulation and transmission module is used to encapsulate the grouped binlog logs into data packets and send the encapsulated binlog logs to the corresponding processing modules through the message queue. The data flow of each group of binlog logs corresponds to a different processing module; The data screening module is used to screen out the data to be detected that meets the preset criteria from the corresponding binlog log based on the preset criteria configured by each processing module; An anomaly detection module, used to analyze whether the data to be detected meets the abnormal condition, and mark the data to be detected that meets the abnormal condition as abnormal data; The alarm generation and notification module is used to generate alarm information based on the abnormal data and send the alarm information to the business processing end.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an abnormal data monitoring program stored in the memory and executable on the processor. When the abnormal data monitoring program is executed by the processor, the steps of the abnormal data monitoring method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores an abnormal data monitoring program, which, when executed by a processor, implements the steps of the abnormal data monitoring method according to any one of claims 1 to 7.

Images