Data anomaly detection method and device, equipment, medium and product

Abstract

The invention discloses a data anomaly detection method, device and equipment, a medium and a product, relates to the technical field of data analysis, and can be applied to the field of financial science and technology. Performing format standardization on the transaction log data output by the application program to obtain standard transaction log data; outputting the standard transaction log data to a distributed message publishing and subscribing system; reading a standard transaction log from the distributed publishing and subscribing message system through a data collection engine, and analyzing the standard transaction log to obtain analyzed log data; and performing anomaly detection on the analyzed log data to obtain abnormal transaction data. The transaction log data are stored in the distributed publishing and subscribing message system after being subjected to standardization processing, and anomaly detection is performed after the transaction log data are analyzed, so that efficient management and real-time monitoring of the transaction log data are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology and can be applied to the field of financial technology, and in particular to a data anomaly detection method, device, equipment, medium and product. Background Art

[0002] As the core of the modern economy, the stability and security of the financial industry's trading system is of vital importance. Financial trading systems process a large amount of transaction data every day, which is not only huge but also complex, involving multiple business types and channels.

[0003] However, the existing transaction monitoring technology has obvious deficiencies in the following aspects. First, the real-time performance is insufficient. Traditional transaction monitoring systems often cannot achieve real-time data collection and analysis, resulting in the inability to timely discover and handle abnormal transactions; second, the data is poorly structured. Due to the different log formats generated by different systems and channels, it is difficult to uniformly process and analyze the data; third, the automation level is low. When parsing and translating business codes, the existing system still relies on a large number of manual operations, which is inefficient and prone to errors; fourth, the monitoring and alarm mechanism is imperfect. The existing monitoring system lacks an intelligent alarm mechanism and cannot issue early warnings for abnormal transactions in the first place, affecting the security and stability of the system. Summary of the invention

[0004] The present invention provides a data anomaly detection method, device, equipment, medium and product to solve the problem of low transaction monitoring efficiency.

[0005] According to one aspect of the present invention, a data anomaly detection method is provided, comprising:

[0006] The format of the transaction log data output by the application is standardized to obtain standard transaction log data; the transaction log data shown includes a transaction number and a channel event number;

[0007] Outputting the standard transaction log data to a distributed publish-subscribe messaging system;

[0008] Reading a standard transaction log from the distributed publish-subscribe message system through a data collection engine, parsing the standard transaction log to obtain parsed log data;

[0009] Anomaly detection is performed on the parsed log data to obtain abnormal transaction data.

[0010] According to another aspect of the present invention, there is provided a data anomaly detection device, comprising:

[0011] The data acquisition module is used to standardize the format of the transaction log data output by the application program to obtain standard transaction log data; the transaction log data includes a transaction number and a channel event number;

[0012] A data publishing module, used for outputting the standard transaction log data to a distributed publish-subscribe messaging system;

[0013] A data parsing module, used to read a standard transaction log from the distributed publish-subscribe message system through a data collection engine, parse the standard transaction log, and obtain parsed log data;

[0014] The detection module is used to perform anomaly detection on the parsed log data to obtain abnormal transaction data.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data anomaly detection method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data anomaly detection method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided. The computer program product comprises a computer program. When the computer program is executed by a processor, the method for detecting anomalies in data according to any one of the embodiments of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention solves the problem of low efficiency of current transaction monitoring by standardizing and storing transaction log data and then performing anomaly detection after parsing the log data, thereby achieving efficient management and real-time monitoring of log data.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a data anomaly detection method provided according to an embodiment of the present invention;

[0025] Figure 2 is a flow chart of a data anomaly detection method adopted in an embodiment of the present invention;

[0026] Figure 3 is a structural schematic diagram of a data anomaly detection device provided according to an embodiment of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device for implementing the data anomaly detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of data involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] Figure 1 This is a flow chart of a data anomaly detection method provided according to an embodiment of the present invention. The present invention is applicable to data anomaly detection, especially to financial scenario transaction data anomaly detection. The method can be performed by a data anomaly detection device provided by an embodiment of the present invention. The data anomaly detection device can be implemented in the form of hardware and / or software and can be integrated into an electronic device that carries a data detection function, such as a server. Figure 1 As shown, the method includes:

[0031] S110. Standardize the format of the transaction log data output by the application to obtain standard transaction log data; the transaction log data includes a transaction number and a channel event number.

[0032] Among them, the transaction log data is the log data generated by the transaction behavior of the application, including the transaction number and the channel event number; the transaction number is used to uniquely identify a transaction in the system; the channel event number is an identifier of a specific event or operation related to the transaction; the application is an application in a financial scenario, and the application in the embodiment of the present invention may be an Industrial and Commercial Bank of China application.

[0033] Specifically, when an application in a financial scenario generates log data for transaction behavior, the corresponding transaction log data is obtained and the format is standardized to obtain standard transaction log data; it should be noted that the specific format standardization can standardize the log output format through a specific log output component, and the embodiment of the present invention does not specifically limit the specific format standardization method.

[0034] It is understandable that by obtaining the transaction number and channel event number in the transaction log data, the complete record of a specific transaction can be quickly found, including detailed information such as transaction time, amount, and parties involved; and combined with the channel event number, the processing flow of the transaction in the system can be further understood, including the timestamp of each stage, processing status, etc., to achieve accurate monitoring of the transaction log data.

[0035] Optionally, the format of the transaction log data output by the application is standardized to obtain standard transaction log data, including: standardizing the format of the transaction log data output by the application through a standardized component to obtain standard transaction log data.

[0036] Specifically, in the application, the log output format is standardized through specific log output components to ensure that specific codes such as transaction numbers and channel event numbers are displayed in a structured form; the transaction log data can be standardized through the data dictionary, and the required transaction numbers and channel event numbers can be extracted and displayed. For example, the transaction number can be keyword: 0001, and its corresponding channel event number is 002.

[0037] It can be understood that standardizing the format of transaction logs based on standardized components ensures the consistency and readability of log data, prevents applications from arbitrarily outputting log data, standardizes the output format of log data corresponding to applications, prevents random printing, simplifies the log parsing and analysis process, and further avoids the problem of difficulty in unified processing and analysis of transaction log data due to different log formats generated by different channels.

[0038] S120: Output the standard transaction log data to the distributed publish-subscribe messaging system.

[0039] Among them, the distributed publish-subscribe messaging system is preferably Kafka, which is an open source stream processing platform developed by the Apache Software Foundation and a high-throughput distributed publish-subscribe messaging system. By outputting standard transaction log data to the distributed publish-subscribe messaging platform, efficient storage of log data is ensured.

[0040] In an optional manner of the embodiment of the present invention, outputting the standard transaction log data to the distributed publish-subscribe messaging system includes: collecting the standard transaction log data in real time through a general log collection and distribution system and transmitting it to the distributed publish-subscribe messaging system.

[0041] The general log collection and distribution system is Fluentd, which can collect logs from multiple data sources, filter and process the logs, and distribute them to multiple storage and processing systems. The application uses fluentd to collect and transmit standard transaction log data in real time to Kafka, realizing efficient transmission and storage of log data.

[0042] S130. Read the standard transaction log from the distributed publish-subscribe message system through a data collection engine, parse the standard transaction log, and obtain parsed log data.

[0043] Among them, the data collection engine is preferably Logstash, which can dynamically unify data from different data sources and standardize the data to the selected destination; parsing log data is to extract log data information of business codes in standard transaction log data.

[0044] Specifically, Logstash is used to read standard transaction logs from Kafka, parse and translate the standard transaction logs, and extract the corresponding business code as parsed log data.

[0045] Optionally, a standard transaction log is read from a distributed publish-subscribe message system through a data collection engine, and the standard transaction log is parsed to obtain parsed log data, including: reading the standard transaction log in segments from the distributed publish-subscribe message system through a data collection engine; and parsing the standard transaction log according to a transaction code dictionary to obtain parsed log data.

[0046] The transaction code dictionary is a predefined specific code dictionary, including transaction numbers and corresponding business codes. Logstash is used to read standard transaction logs from Kafka in segments, and the business codes in the standard transaction logs are automatically parsed and translated according to the predefined specific code dictionary, and the parsed and translated business codes are used as parsed log data.

[0047] It is understandable that using Logstash to read standard transaction logs from Kafka in segments and parse them based on a preset specific code dictionary can improve parsing accuracy and efficiency, and effectively avoid the situation where the existing system still relies on a large amount of manual operations when parsing and translating business codes, which is inefficient and error-prone.

[0048] In an optional manner of the embodiment of the present invention, data in the predefined specific code dictionary can be updated online in real time, and the updated dictionary data can be positively fed back to the running parser.

[0049] Specifically, the data update method in a specific code dictionary can be performed by manual modification by technicians on the back-end server, or by real-time update based on real-time communication interaction between the front and back ends of the server; it should be noted that the specific real-time update method can be updated by relevant technicians according to specific actual needs, and the present invention does not specifically limit this.

[0050] It is understandable that by updating the data in a specific code dictionary in real time, not only can the accuracy and efficiency of code parsing be improved, but it can also facilitate the maintenance and management of dictionary data by relevant technical personnel.

[0051] S140: Perform anomaly detection on the parsed log data to obtain abnormal transaction data.

[0052] Among them, abnormal transaction data refers to the number of transaction log data within a certain period of time or business indicators within the abnormal range threshold; anomaly detection can be achieved by analyzing the transaction number and channel event number of the log data and other business codes through model identification, or by setting thresholds for judgment.

[0053] The parsed log data is input into the model for anomaly detection to identify potential abnormal transaction data; thus achieving real-time monitoring of financial transaction data.

[0054] Optionally, perform anomaly detection on the parsed log data to obtain abnormal transaction data, including:

[0055] Based on the transaction anomaly rules, anomaly detection is performed on the parsed log data to obtain abnormal transaction data.

[0056] Among them, the transaction anomaly rule is a pre-set transaction anomaly threshold. It should be noted that relevant users can set reasonable monitoring rules and thresholds according to business logic and actual conditions, and the embodiment of the present invention does not specifically limit this.

[0057] It can be understood that the transaction anomaly rule can be the frequency of obtaining parsed log data. Based on the pre-set transaction anomaly rule, the parsed log data is detected for anomalies to obtain abnormal transaction data; for example: the transaction anomaly rule is set to obtain parsed log data 100 times in 1 minute; if the frequency of parsed log data obtained in the actual one minute is 10 times, a threshold prompt is given, and the current parsed log data is abnormal transaction data.

[0058] In an optional manner of the embodiment of the present invention, after obtaining the abnormal transaction data, it includes: sending an alarm notification via email or text message.

[0059] When abnormal transaction data is detected, an alarm notification is sent via email, SMS or other means, and the alarm notification can carry information related to the current abnormal transaction data so that technical personnel can take corresponding measures based on the abnormal transaction data.

[0060] It is understandable that by sending an alarm notification after detecting abnormal transaction data and improving the monitoring and alarm mechanism of transaction log data, it is possible to avoid the existing monitoring system being unable to issue early warnings for abnormal transactions in the first place, which may affect the security and stability of applications in financial scenarios.

[0061] The embodiment of the present invention standardizes the format of transaction log data output by an application to obtain standard transaction log data; the transaction log data includes a transaction number and a channel event number; the standard transaction log data is output to a distributed publish-subscribe message system; the standard transaction log data is read from the distributed publish-subscribe message system by a data collection engine, and the standard transaction log data is parsed to obtain parsed log data; anomaly detection is performed on the parsed log data to obtain abnormal transaction data; the above technical solution solves the problem of low transaction monitoring efficiency, by standardizing the transaction log data and storing it in a distributed publish-subscribe message system, and performing abnormal transaction data detection after parsing, thereby achieving efficient management and real-time monitoring of transaction logs.

[0062] Optionally, after obtaining the parsed log data, the method further includes: storing the parsed log data in a search and data analysis engine cluster.

[0063] Among them, the search and data analysis engine cluster is the Elasticsearch cluster. Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine.

[0064] The parsed and translated log data is stored in the Elasticsearch cluster through Logstash, ensuring efficient storage and fast retrieval of parsed log data.

[0065] Optionally, after storing the parsed log data in the search and data analysis engine cluster, the method further includes: obtaining the parsed log data from the search and data analysis engine cluster by using a visualization tool; and visualizing the parsed log data.

[0066] Among them, the visualization tool is preferably Grafana, which is a data monitoring and visualization tool, which is connected to the Elasticsearch cluster; the parsed log data visualization can be displayed through a curve chart, a pie chart or a table.

[0067] Specifically, the visualization tool is connected to the search and data analysis engine cluster, and the parsed log data stored in the search and data analysis engine cluster is obtained. A real-time monitoring dashboard is created based on the parsed log data, and key business indicators and log data are displayed through curve charts, pie charts or tables.

[0068] It is understandable that creating real-time monitoring dashboards through the Grafana visualization tool to display key business indicators and log data improves the reliability and maintainability of the system.

[0069] Accordingly, based on the transaction anomaly rules, anomaly detection is performed on the parsed log data, and the abnormal transaction data obtained can be:

[0070] The transaction anomaly rule can be the range threshold in the curve graph, which is the upper and lower limits in the curve graph. Based on the data points in the curve graph, the size of each data point is compared with the preset range threshold, and the data points exceeding the threshold are recorded to obtain the abnormal transaction data related information corresponding to the data point.

[0071] It is understandable that by setting transaction anomaly rules and visually displaying key business indicators and log data through curve charts, pie charts or tables, it is helpful to intuitively and quickly identify abnormal transaction data, thereby improving the efficient management and real-time monitoring of transaction log data.

[0072] The embodiment of the present invention parses the storage of log data through a search and data analysis engine cluster, thereby ensuring efficient transmission, storage and retrieval of log data; and uses visualization tools for real-time monitoring and alarming, thereby improving the reliability and maintainability of the system.

[0073] like Figure 2A flowchart of a data anomaly detection method is provided, which obtains transaction log data related to application process 1, standardizes it based on standardized components, obtains corresponding standard transaction log data, and collects it in real time to Kafka through fluentd for subsequent transmission; uses Logstash to read standard transaction log data from Kafka in segments, and parses and translates business codes in standard transaction log data online based on predefined data dictionary codes; and stores the parsed log data in an Elasticsearch cluster, creates a real-time monitoring dashboard through visualization tools to achieve real-time monitoring of parsed log data, and performs alarm processing when an abnormal situation is detected, and sends alarm notifications of abnormal transaction data corresponding to the abnormal situation through email, SMS or other methods.

[0074] It can be understood that in the embodiment of the present invention, the logs are standardized and stored in the Elasticsearch cluster so as to perform automated abnormal transaction monitoring based on the Elasticsearch cluster. This can be applied to various scenarios that require efficient management and monitoring of log data, thereby improving the reliability and maintainability of transaction data.

[0075] Figure 3 1 is a schematic diagram of the structure of a data anomaly detection device provided according to an embodiment of the present invention. The present invention can be applied to data anomaly detection, especially to financial scenario transaction data anomaly detection. The data anomaly detection device can be implemented in the form of hardware and / or software, and can be integrated into an electronic device carrying a data detection function, such as a server. Figure 3 As shown, the data anomaly detection device 200 includes a data acquisition module 210, a data publishing module 220, a data analysis module 230 and a detection module 240.

[0076] The data acquisition module 210 is used to standardize the format of the transaction log data output by the application program to obtain standard transaction log data; the transaction log data shown includes a transaction number and a channel event number;

[0077] Data publishing module 220, used to output standard transaction log data to a distributed publish-subscribe messaging system;

[0078] The data analysis module 230 is used to read the standard transaction log from the distributed publish-subscribe message system through the data collection engine, analyze the standard transaction log, and obtain the analyzed log data;

[0079] The detection module 240 is used to perform anomaly detection on the parsed log data to obtain abnormal transaction data.

[0080] The embodiment of the present invention standardizes the format of transaction log data output by an application to obtain standard transaction log data; the transaction log data includes a transaction number and a channel event number; the standard transaction log data is output to a distributed publish-subscribe message system; the standard transaction log data is read from the distributed publish-subscribe message system by a data collection engine, and the standard transaction log data is parsed to obtain parsed log data; anomaly detection is performed on the parsed log data to obtain abnormal transaction data; the above technical solution solves the problem of low transaction monitoring efficiency, by standardizing the transaction log data and storing it in a distributed publish-subscribe message system, and performing abnormal transaction data detection after parsing, thereby achieving efficient management and real-time monitoring of transaction logs.

[0081] Optionally, the data acquisition module 210 is specifically used to standardize the format of the transaction log data output by the application program through a standardized component to obtain standard transaction log data.

[0082] Optionally, the data parsing module 230 is specifically used to read the standard transaction log in segments from the distributed publish-subscribe message system through the data collection engine; parse the standard transaction log according to the transaction code dictionary to obtain parsed log data.

[0083] Optionally, the anomaly detection device further includes a parsing log data storage module, which is used to store the parsing log data in a search and data analysis engine cluster.

[0084] Optionally, the detection module 240 is specifically configured to perform anomaly detection on the parsed log data based on anomaly transaction rules to obtain abnormal transaction data.

[0085] Optionally, the data anomaly detection device also includes a parsing log data acquisition module, which is used to obtain parsing log data from the search and data analysis engine cluster through a visualization tool; and visualize the parsing log data.

[0086] The data anomaly detection device provided in the embodiment of the present invention can execute the data anomaly detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0087] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0088] Figure 41 is a schematic diagram of the structure of an electronic device 10 for implementing the data anomaly detection method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0089] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0090] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0091] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data anomaly detection method.

[0092] In some embodiments, the data anomaly detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data anomaly detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data anomaly detection method in any other appropriate manner (e.g., by means of firmware).

[0093] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0095] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0098] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0100] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data anomaly detection method, characterized in that: include: The format of the transaction log data output by the application is standardized to obtain standard transaction log data; the transaction log data shown includes a transaction number and a channel event number; Outputting the standard transaction log data to a distributed publish-subscribe messaging system; Reading a standard transaction log from the distributed publish-subscribe message system through a data collection engine, parsing the standard transaction log to obtain parsed log data; Anomaly detection is performed on the parsed log data to obtain abnormal transaction data.

2. The method according to claim 1, characterized in that: The format of the transaction log data output by the application is standardized to obtain standard transaction log data, including: Through the standardized component, the format of the transaction log data output by the application is standardized to obtain standard transaction log data.

3. The method according to claim 1, characterized in that: The method of reading the standard transaction log from the distributed publish-subscribe message system through the data collection engine and parsing the standard transaction log to obtain parsed log data includes: Reading standard transaction logs in segments from the distributed publish-subscribe messaging system through a data collection engine; The standard transaction log is parsed according to the transaction code dictionary to obtain parsed log data.

4. The method according to claim 1, characterized in that: After obtaining the parsed log data, it also includes: The parsed log data is stored in a search and data analysis engine cluster.

5. The method according to claim 1, characterized in that Perform anomaly detection on the parsed log data to obtain abnormal transaction data, including: Based on the transaction anomaly rules, anomaly detection is performed on the parsed log data to obtain abnormal transaction data.

6. The method according to claim 4, characterized in that The method further comprises: Obtaining the parsed log data from the search and data analysis engine cluster through a visualization tool; The parsed log data is visualized.

7. A data anomaly detection device, characterized in that: include: The data acquisition module is used to standardize the format of the transaction log data output by the application program to obtain standard transaction log data; the transaction log data includes a transaction number and a channel event number; A data publishing module, used for outputting the standard transaction log data to a distributed publish-subscribe messaging system; A data parsing module, used to read a standard transaction log from the distributed publish-subscribe message system through a data collection engine, parse the standard transaction log, and obtain parsed log data; The detection module is used to perform anomaly detection on the parsed log data to obtain abnormal transaction data.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data anomaly detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data anomaly detection method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data anomaly detection method according to any one of claims 1 to 6.

Images