Data anomaly detection method and device, electronic equipment, medium and program product

Abstract

The invention discloses a data anomaly detection method and device, electronic equipment, a medium and a program product, and the method comprises the steps: obtaining to-be-detected data which comprises at least one data format; based on an SPI mechanism, target anomaly detection algorithms corresponding to the to-be-detected data are determined, and the target anomaly detection algorithms comprise at least two anomaly detection algorithms; and detecting the to-be-detected data through the target anomaly detection algorithm to obtain an anomaly detection result. According to the data anomaly detection method and device, the requirement of a data anomaly detection task with high real-time performance requirement can be met, anomaly detection of multi-source data is supported, an anomaly detection algorithm can be dynamically expanded, the detection efficiency and the anomaly detection accuracy can be improved, the anomaly detection result can be processed in time, and the use experience of a user is improved.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of big data processing technology, and in particular to a data anomaly detection method, device, electronic device, medium, and program product. Background Art

[0002] With the rapid development of the Internet, big data analysis has become a hot research field. In big data analysis, rich data assets can provide strong information support for people's analysis and decision-making, but various data quality problems may arise during data collection and processing. How to detect data anomalies is of great significance.

[0003] In related technologies, data anomalies are detected by data miners discovering data problems during the use of data and reporting the problems to data quality management personnel, who then correct the abnormal data. This manual feedback method, on the one hand, cannot achieve timely data detection and has poor real-time performance. On the other hand, it consumes a lot of time and increases the difficulty of data error correction, resulting in the inability to process abnormal data in a timely manner, which reduces the value of data and affects the user experience. Summary of the invention

[0004] In view of this, the present application provides a data anomaly detection method, device, electronic device, medium and program product, which supports anomaly detection of multi-source data, and the anomaly detection algorithm can be dynamically expanded, which can improve the detection efficiency and accuracy of anomaly detection and enhance the user experience.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, the present application provides a data anomaly detection method, comprising: obtaining data to be detected, wherein the data to be detected includes at least one data format; based on the SPI mechanism, determining a target anomaly detection algorithm corresponding to the data to be detected, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms; detecting the data to be detected by using the target anomaly detection algorithm to obtain an anomaly detection result.

[0007] In some embodiments, the data to be detected is detected by a target anomaly detection algorithm to obtain an anomaly detection result, including: based on a distributed memory computing framework, data preprocessing is performed on the training data to obtain a data sample set; the data sample set is sent to each computing node of the distributed memory computing framework, and a target model corresponding to the target anomaly detection algorithm is determined through the computing node; the data to be detected is detected through the target model to obtain an anomaly detection result.

[0008] In some embodiments, the target model includes at least two anomaly detection models; the target model is used to detect the data to be detected to obtain an anomaly detection result, including: detecting the data to be detected by at least two anomaly detection models respectively to obtain a prediction result corresponding to each anomaly detection model; and performing weighted average calculation on the prediction results to obtain an anomaly detection result.

[0009] In some embodiments, the target model includes at least two anomaly detection models; the target model is used to detect the data to be detected to obtain an anomaly detection result, including: detecting the data to be detected by the first N-1 anomaly detection models of at least two anomaly detection models to obtain a prediction result corresponding to each anomaly detection model, wherein N is a positive integer greater than 1; using the detection result as an input of the Nth anomaly detection model of at least two anomaly detection models, and performing prediction by the Nth anomaly detection model to obtain an anomaly detection result.

[0010] In some embodiments, based on a distributed memory computing framework, data preprocessing is performed on training data to obtain a data sample set, including: performing data preprocessing on the training data according to the type of data through the distributed memory computing framework to obtain processed training data; converting the processed training data into a data format supported by the distributed memory computing framework to obtain a data sample set.

[0011] In some embodiments, based on the SPI mechanism, a target anomaly detection algorithm corresponding to the data to be detected is determined, including: according to the data format of the data to be detected, at least two anomaly detection algorithms corresponding to the data to be detected are determined through a configuration file of the SPI mechanism, wherein the configuration file includes a mapping relationship between different data formats and anomaly detection algorithms; based on at least two anomaly detection algorithms, a target anomaly detection algorithm is determined.

[0012] In some embodiments, after the data to be detected is detected by a target anomaly detection algorithm to obtain an anomaly detection result, the above method also includes: obtaining audit rules and audit indicators corresponding to the data to be detected; analyzing the anomaly detection results through the audit rules and audit indicators to obtain a data quality report corresponding to the data to be detected.

[0013] In a second aspect, the present application provides a data anomaly detection device, comprising: a first acquisition module, used to acquire data to be detected, wherein the data to be detected includes at least one data format; a determination module, based on the SPI mechanism, determines a target anomaly detection algorithm corresponding to the data to be detected, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms; a detection module, which detects the data to be detected through the target anomaly detection algorithm to obtain an anomaly detection result.

[0014] In some embodiments, the detection module includes: a first determination unit, used to perform data preprocessing on the training data based on a distributed memory computing framework to obtain a data sample set; a second determination unit, used to send the data sample set to each computing node of the distributed memory computing framework, and determine the target model corresponding to the target anomaly detection algorithm through the computing node; a detection unit, used to detect the data to be detected through the target model to obtain anomaly detection results.

[0015] In some embodiments, the target model includes at least two anomaly detection models; the detection unit is used to perform the following steps: respectively detecting the data to be detected by at least two anomaly detection models to obtain the prediction results corresponding to each anomaly detection model; performing weighted average calculation on the prediction results to obtain the anomaly detection results.

[0016] In some embodiments, the target model includes at least two anomaly detection models; a detection unit is used to perform the following steps: detecting the data to be detected by using the first N-1 anomaly detection models of the at least two anomaly detection models to obtain a prediction result corresponding to each anomaly detection model, wherein N is a positive integer greater than 1; using the detection result as an input of the Nth anomaly detection model of the at least two anomaly detection models, and performing prediction by the Nth anomaly detection model to obtain an anomaly detection result.

[0017] In some embodiments, the first determination unit is used to perform the following steps: preprocess the training data according to the data type through a distributed memory computing framework to obtain processed training data; convert the processed training data into a data format supported by the distributed memory computing framework to obtain a data sample set.

[0018] In some embodiments, a determination module is used to perform the following steps: determine at least two anomaly detection algorithms corresponding to the data to be detected according to the data format of the data to be detected through a configuration file of the SPI mechanism, wherein the configuration file includes a mapping relationship between different data formats and anomaly detection algorithms; and determine a target anomaly detection algorithm based on at least two anomaly detection algorithms.

[0019] In some embodiments, the above-mentioned device also includes: a second acquisition module, which is used to acquire the audit rules and audit indicators corresponding to the data to be detected after the data to be detected is detected by using the target anomaly detection algorithm and the anomaly detection result is obtained; an analysis module, which is used to analyze the anomaly detection result through the audit rules and audit indicators to obtain a data quality report corresponding to the data to be detected.

[0020] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, some or all of the steps in the above method are implemented.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when the computer program is executed by a processor.

[0022] In a fifth aspect, the present application provides a computer program product, including a computer program or instructions, which implement some or all of the steps in the above method when executed by a processor.

[0023] In a sixth aspect, the present application provides a computer program, comprising a computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes some or all of the steps for implementing the above method.

[0024] In the present application, by obtaining data to be detected, wherein the data to be detected includes at least one data format, based on the SPI mechanism, a target anomaly detection algorithm corresponding to the data to be detected is determined, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms, and the data to be detected is detected by the target anomaly detection algorithm to obtain an anomaly detection result. It is possible to automatically determine the target anomaly detection algorithm corresponding to the data to be detected in different data formats, and perform an anomaly analysis on the data to be detected through the target anomaly detection algorithm, thereby meeting the needs of data anomaly detection tasks with high real-time requirements, supporting anomaly detection of multi-source data, and the anomaly detection algorithm can be dynamically expanded, which is conducive to improving the detection efficiency and the accuracy of anomaly detection, so as to process the anomaly detection results in time and improve the user experience.

[0025] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0027] Figure 1 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0028] Figure 2 It is a structural schematic diagram of a data anomaly detection device provided in an embodiment of the present application;

[0029] Figure 3is a schematic diagram of a data anomaly detection system provided in an embodiment of the present application;

[0030] Figure 4 This is a schematic diagram of an implementation process of the data anomaly detection method provided in an embodiment of the present application;

[0031] Figure 5 This is a schematic diagram of an implementation method of the SPI mechanism provided in an embodiment of the present application;

[0032] Figure 6 This is a schematic diagram of an implementation process of the isolation forest algorithm provided in an embodiment of the present application;

[0033] Figure 7 This is another implementation flow diagram of the data anomaly detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following is a description of the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. In the following description, reference is made to the drawings that form a part of the present application and show the specific aspects of the embodiments of the present application or the specific aspects of the embodiments of the present application in an illustrative manner. It should be understood that the embodiments of the present application can be used in other aspects and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present application is defined by the appended claims. For example, it should be understood that the disclosure of the described method can be equally applicable to the corresponding device or system capable of performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units such as functional units to perform the described one or more method steps (for example, one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific device is described based on one or more units such as functional units, the corresponding method may include a step to perform the functionality of the one or more units (e.g., a step to perform the functionality of the one or more units, or multiple steps, each of which performs the functionality of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Further, it should be understood that, unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0035] With the rapid development of the Internet and the Internet of Things, the speed of big data generation has exploded. Data has been regarded as an important strategic asset by enterprises, and big data analysis has become a hot research field in the industry and academia. In big data analysis, rich data assets can provide strong information support for people's analysis and decision-making. However, in the process of data collection and processing, data value missing or abnormal, data attribute loss or redundancy and other data quality problems may occur due to input errors, human tampering and mechanical failure. The existence of these abnormal data may mislead the results of data analysis and decision-making. Therefore, it is of great significance to scientifically and rationally control data quality and detect data anomalies.

[0036] Data anomaly detection is an important part of data analysis and data science. It is an ongoing process that needs to be performed regularly to respond to data changes and new business needs. Through effective anomaly detection, the accuracy and reliability of data can be ensured, providing a solid foundation for data analysis and decision-making. It involves identifying and processing data points that are significantly different from most data points in the data set. These abnormal data points may be caused by errors, fraud, system failures or other abnormal situations.

[0037] In the field of data anomaly detection, related technology 1 provides a method and device for log data anomaly detection, by clustering the original logs according to log similarity to obtain several categories of logs; counting the log volume per unit time of each category of logs within a preset time before the current detection moment, to obtain the time series log volume data of each category of logs; judging whether the current detection moment is an abnormal point based on the time series log volume data of each category of logs and the corresponding dynamic threshold. This solution is to discover log anomalies from the log volume level, and achieve the purpose of detecting abnormal events by detecting changes in log volume.

[0038] Related technology 2 provides a method for detecting abnormal user access behavior based on isolation forests, and proposes a machine learning algorithm based on isolation forests to improve the shortcomings of the current technology, such as the difficulty in describing artificial features and the high labor cost. At the same time, it can automatically extract new abnormal behavior features for the isolation forest model to learn by itself, further improving the robustness of the isolation forest model. In addition, in the isolation forest model, the construction of each isolated binary tree does not affect each other, and under the balance of weight values, the isolation forest model has higher detection accuracy and efficiency for high-dimensional data.

[0039] However, the above-mentioned related technologies only provide a data source quality anomaly detection in a specific application scenario, and cannot cope with scenarios with multiple types of data sources, especially in big data and massive data, and cannot meet the data auditing needs with high real-time requirements. For example, related technology 1 is only for anomaly detection of log data, and is not compatible with different types of data sources. For different data sources, corresponding audit scheduling modules need to be developed to achieve data anomaly detection. It is not flexible and lacks a unified audit scheduling module. For another example, although related technology 2 improves the anomaly detection effect to a certain extent, it ignores the differences in the anomaly detection capabilities between isolated binary trees and the fact that it takes a lot of time to build a large number of isolated binary trees under large-scale data. The robustness and effectiveness of the algorithm need to be further improved.

[0040] Based on the above problems, the present application proposes a data anomaly detection method, which obtains data to be detected, wherein the data to be detected includes at least one data format, and determines a target anomaly detection algorithm corresponding to the data to be detected based on the SPI mechanism, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms, and detects the data to be detected through the target anomaly detection algorithm to obtain an anomaly detection result. The method can automatically determine the target anomaly detection algorithm corresponding to the data to be detected in different data formats, and perform an anomaly analysis on the data to be detected through the target anomaly detection algorithm, thereby meeting the needs of data anomaly detection tasks with high real-time requirements, supporting anomaly detection of multi-source data, and the anomaly detection algorithm can be dynamically expanded, which is conducive to improving the detection efficiency and the accuracy of anomaly detection, so as to process the anomaly detection results in time and improve the user experience.

[0041] In some embodiments, the method described in the embodiments of the present application can be applied to scenarios such as data anomaly detection, data verification, and data quality monitoring. In some embodiments, the method described in the embodiments of the present application can also be applied to cloud computing, big data edge computing, education, medical care, social networking, entertainment, online classes, office work and other fields.

[0042] The following describes an exemplary application of the electronic device provided in the embodiment of the present application. The electronic device provided in the embodiment of the present application can be a laptop, a tablet computer, a desktop computer, a mobile device (e.g., a mobile phone, a wearable smart watch, a dedicated messaging device), an electric car, an electric bicycle, and other rechargeable devices, but is not limited thereto. Alternatively, the electronic device can also be implemented as a server.

[0043] In some embodiments, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms, but is not limited thereto. In some embodiments, the server and the electronic device may be directly or indirectly connected via wired or wireless communication, and this is not specifically limited in the embodiments of the present application.

[0044] See also Figure 1 , Figure 1 is a structural schematic diagram of an electronic device provided in an embodiment of the present application, Figure 1 The electronic device 100 shown includes: at least one processor 110, a memory 150, at least one network interface 120 and a user interface 130. The various components in the electronic device 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140 is not described in detail. Figure 1 Various buses are labeled as bus system 140 .

[0045] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The user interface 130 includes one or more output devices 131 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 130 also includes one or more input devices 132, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0047] The memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 150 may optionally include one or more storage devices that are physically remote from the processor 110.

[0048] The memory 150 includes a volatile memory or a nonvolatile memory, and may also include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 150 described in the embodiments of the present application is intended to include any suitable type of memory.

[0049] In some embodiments, the memory 150 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0050] The operating system 151 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0051] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120. Exemplary network interfaces 120 include: Bluetooth, WiFi, and universal serial bus (USB).

[0052] The presentation module 153 is used to enable presentation of information (eg, a user interface for operating peripheral devices and displaying content and information) via one or more output devices 131 (eg, a display screen, a speaker, etc.) associated with the user interface 130 .

[0053] The input processing module 154 is configured to detect one or more user inputs or interactions from one of the one or more input devices 132 and to translate the detected inputs or interactions.

[0054] In some embodiments, the data anomaly detection method provided in the embodiments of the present application can be implemented in software and stored in the memory 150. Figure 2 , Figure 2 15 is a structural diagram of a data anomaly detection device provided in an embodiment of the present application, which may be software in the form of a program and a plug-in, etc. The data anomaly detection device 155 includes the following software modules: a first acquisition module 1551, a determination module 1552, and a detection module 1553. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0055] In other embodiments, the data anomaly detection device 155 provided in the embodiment of the present application can be implemented in hardware. As an example, the data anomaly detection device 155 provided in the embodiment of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data anomaly detection method provided in the embodiment of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA) or other electronic components.

[0056] The data anomaly detection method provided in the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0057] In some embodiments, the data anomaly detection method provided in the embodiments of the present application can be applied to a data anomaly detection system, and the data anomaly detection system can be used for data anomaly detection.

[0058] See also Figure 3 , Figure 3 300 is a structural diagram of a data anomaly detection system provided in an embodiment of the present application. The data anomaly detection system 300 can be deployed in an electronic device, and may include: a data source configuration module 301, a target anomaly detection algorithm determination module 302, a distributed unified audit scheduling module 303, and a data quality report generation module 304. The following describes each module included in the data anomaly detection system 300.

[0059] 1. Data source configuration module 301:

[0060] The data source configuration module 301 can obtain a variety of different types of data sources and can support multi-source data, such as Figure 3 The JavaScript object notation (JSON) file, parquet file, hive data, mysql data shown in the text may also be an avro file, a sequence file, etc., which is not specifically limited in the embodiments of the present application.

[0061] Here, JSON file is a lightweight data exchange format that stores and represents data in a text format independent of programming languages. Parquet file is a columnar storage file format, mainly used in Hadoop ecosystem, where Hadoop is a distributed system infrastructure; Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop; MySQL data is data in a relational database management system; Avro file is a binary-based file format, which belongs to a row storage format, storing data in files by row, and is used to efficiently store and transmit large amounts of data; Sequencefile is a flat file designed for Hadoop to store binary key-value pairs.

[0062] 2. Target anomaly detection algorithm determination module 302:

[0063] For different types of data sources or data format files, the target anomaly detection algorithm determination module 302 supports dynamic expansion of different anomaly detection algorithms, and can dynamically select corresponding target anomaly detection algorithms for different types of data sources or data format files.

[0064] The target anomaly detection algorithm can be determined by obtaining user input data (recorded as data to be detected) from different data sources, and dynamically loading the target anomaly detection algorithm corresponding to the user input data using a service provider interface (SPI) mechanism according to the power law distribution, normal distribution, and data dimension (e.g., single dimension, multi-dimensional) of the user input data. After the target anomaly detection algorithm is determined, the user input data is detected through the distributed unified audit scheduling module 303.

[0065] Here, the Java SPI mechanism is a service discovery mechanism used to implement extension points of a framework or library, allowing component implementations to be dynamically inserted or replaced at runtime.

[0066] In some embodiments, a switch may be provided in the target anomaly detection algorithm determination module 302, through which the module may be turned on or off, so that the target anomaly detection algorithm determination module 302 is easy to manage and plug-and-play.

[0067] In some embodiments, developers can dynamically expand different anomaly detection algorithms (which can be called service providers) in the target anomaly detection algorithm determination module 302 in advance according to changes in data source types and new business needs, such as: anomaly detection autoencoders method, K-nearest neighbor (KNN) method, etc. In this way, the target anomaly detection algorithm determination module 302 can dynamically select target anomaly detection algorithms for multiple different types of data sources to achieve the effect of a data anomaly detection process that is compatible with different types of data sources.

[0068] In some embodiments, the target anomaly detection algorithm can be determined from statistical-based methods, machine learning-based methods, and other anomaly detection algorithms, which may include one or more anomaly detection algorithms, and the anomaly detection algorithm can be dynamically expanded, that is, the target anomaly detection algorithm determination module 302 supports a one-to-one (one data format corresponds to one anomaly detection algorithm) or one-to-many (one data format corresponds to multiple anomaly detection algorithms) mode.

[0069] The following is an introduction to statistical methods and machine learning methods:

[0070] (1) Statistical-based methods.

[0071] a. Interval determination: By setting a threshold interval, if the data point falls outside this interval, it is judged as abnormal. For example, in time series analysis, such an interval can be constructed by comparing the residuals of the predicted value and the actual observed value.

[0072] b. Binary judgment: Similar to interval judgment, but more stringent. It is usually used in binary classification problems to distinguish between normal and abnormal by setting a specific threshold.

[0073] c. Interquartile range method: Use the quartiles (Q1 and Q3) and interquartile range (IQR = Q3-Q1) of the data to identify outliers. Generally, data points below Q1-1.5*IQR or above Q3+1.5*IQR are considered abnormal.

[0074] d. Standard Deviation Method: Assuming that the data follows a normal distribution, the number of standard deviations of a data point from the mean can be used to determine anomalies. In the commonly used three sigma rule, points that are more than 3 standard deviations away from the mean are identified as outliers.

[0075] (2) Methods based on machine learning.

[0076] e. Isolation forest (iForest): It is a tree-based unsupervised learning algorithm used to identify outliers in data. It "isolates" data points by randomly selecting features and split points. Outliers are usually easier to be isolated.

[0077] f. Density-based spatial clustering of applications with noise (DBSCAN): This is a density-based clustering algorithm that can identify core points, boundary points, and noise points (i.e., outliers) in high-density areas.

[0078] g. Local outlier factor (LOF): It is a density-based anomaly detection method that calculates the local density deviation of each data point from its neighbors. Points with significantly lower density than their neighbors are considered anomalies.

[0079] h. One-class support vector machine (One-Class SVM): The core idea of ​​this algorithm is to find a hyperplane that can contain most of the data points (called "normal" data), while excluding abnormal data points and identifying data points that deviate from these patterns.

[0080] 3. Distributed unified audit scheduling module 303:

[0081] The distributed unified audit scheduling module 303 adopts a distributed memory computing framework (such as Spark, Flink, etc.), which can support various types of data sources and solve the problem of repeated development of audit scheduling modules for various types of data sources, thereby reducing development work.

[0082] In some embodiments, the distributed memory computing framework can audit incremental data based on data file timestamps or dynamic data streams, and can also analyze massive historical data to meet some real-time requirements and the need to generate audit results reports in seconds.

[0083] In some embodiments, according to different business needs, developers can use programming languages, such as structured query language (SQL) statements, to write implementation rules corresponding to different anomaly detection algorithms to perform data anomaly detection on the data to be detected, and these anomaly detection algorithms can be uniformly scheduled through the distributed unified audit scheduling module 303. In this way, different target anomaly detection algorithms can be used for different data formats in different application scenarios, so that the data anomaly detection process has high applicability in practical applications.

[0084] In some embodiments, the distributed unified audit scheduling module 303 can pre-process different types of data. The following are some data pre-processing strategies:

[0085] (1) Text data processing.

[0086] Word segmentation: Break the text into words or phrases. Usually, for text data, word segmentation can better extract features.

[0087] Stemming: removing the endings of words to reduce vocabulary diversity.

[0088] Lemmatization: restoring words to their dictionary form to reduce vocabulary diversity.

[0089] (2) Category data processing.

[0090] Encoding: Convert text category labels into numeric labels, such as using one-hot encoding or label encoding.

[0091] (3) Time series data processing.

[0092] Time Series Decomposition: Decompose time series data into trend, seasonality, and random components.

[0093] In some embodiments, the distributed unified audit scheduling module 303 can convert the data obtained after data preprocessing into a format supported by a distributed memory computing framework (such as Spark, Flink, etc.) (such as a data frame (DataFrame) format supported by Spark) to process the data in the converted format. In this way, the differences between the underlying different data sources can be shielded.

[0094] Here, DataFrame is a tabular data structure, similar to an Excel or SQL table, with row labels (index) and column labels (columns). The data type of each column can be different, such as string, integer, or floating point.

[0095] 4. Data quality report generation module 304:

[0096] After obtaining the anomaly detection results corresponding to different types of data sources through the target anomaly detection algorithm determination module 302 and the distributed unified audit scheduling module 303, the data quality report generation module 304 further analyzes the anomaly detection results according to the audit rules and audit indicators to obtain a data quality report.

[0097] In some embodiments, in practical applications, it may be necessary to combine multiple anomaly detection methods to improve the accuracy and robustness of anomaly detection. The target anomaly detection algorithm determination module 302 can select one or more anomaly detection methods for data anomaly detection. The selection of which anomaly detection method can depend on the characteristics of the data to be detected, domain knowledge, the type of outliers, and the required detection performance, etc. For example, a statistical method can be used for preliminary screening, and then a machine learning algorithm can be applied for further identification and classification. Therefore, each batch of task scheduling can generate one or more anomaly detection results. The data quality report generation module 304 combines audit rules and audit indicators to comprehensively analyze multiple anomaly detection results, which can improve the accuracy of anomaly detection to obtain the final data quality report. Compared with a single anomaly detection, the accuracy and robustness of data anomaly detection can be improved, and the quality of data quality reports can be improved.

[0098] The data anomaly detection method provided in the embodiment of the present application is described below in conjunction with the above-mentioned data anomaly detection system.

[0099] Figure 4 Schematic diagram of an implementation flow of the data anomaly detection method provided in the embodiment of the present application. Figure 4 As shown, the data anomaly detection method of the embodiment of the present application may include steps S410 to S430.

[0100] In step S410, data to be detected is obtained.

[0101] The data to be detected includes at least one data format.

[0102] Here, the data to be detected can be data input by a user and includes at least one data format. The data format can be txt, JSON, comma-separated values ​​(CSV), binary, hypertext markup language (HTML), etc., which is not specifically limited in the embodiments of the present application.

[0103] It is understandable that in order to better manage data, before performing data anomaly detection, it is necessary to obtain the data to be detected.

[0104] In some embodiments, the data to be detected in step S410 can be obtained through the above-mentioned data source configuration module 301.

[0105] In some embodiments, the data to be detected in step S410 may be massive data, incremental data, etc., which is not specifically limited in the embodiments of the present application.

[0106] In step S420, based on the SPI mechanism, a target anomaly detection algorithm corresponding to the data to be detected is determined.

[0107] The target anomaly detection algorithm includes at least two anomaly detection algorithms. The target anomaly detection algorithm is used to implement anomaly detection on the data to be detected.

[0108] Here, the SPI mechanism is a service discovery mechanism, which is mainly used to achieve loose coupling and pluggable scalability between modules. It allows applications to dynamically discover and load specific service implementations at runtime. Developers can define an interface and call different anomaly detection algorithms through the interface. Electronic devices can load different anomaly detection algorithms through the SPI mechanism without modifying the core code, which can improve flexibility and scalability.

[0109] In some embodiments, for step S420, according to the data format corresponding to the data to be detected, based on the SPI mechanism, a target anomaly detection algorithm corresponding to the data format of the data to be detected can be determined.

[0110] In some embodiments, the target anomaly detection algorithm in step S420 may be determined by the target anomaly detection algorithm determination module 302 .

[0111] In some embodiments, for step S420, a target anomaly detection algorithm corresponding to the data to be detected in different data formats can be constructed in advance, and the mapping relationship between each data format and the target anomaly detection algorithm matching the data format is stored in the configuration file of the SPI mechanism. When it is necessary to determine the target anomaly detection algorithm, based on the data format of the data to be detected, the target anomaly detection algorithm matching the data format is determined through the mapping relationship stored in the configuration file.

[0112] In some embodiments, for step S420, the determination rules of the target anomaly detection algorithm for the data to be detected in different data formats may be predetermined, for example, data characteristics, data distribution, abnormal value types, etc., and the determination rules may be stored in the configuration file of the SPI mechanism. When it is necessary to determine the target anomaly detection algorithm, based on the data format of the data to be detected, the target anomaly detection algorithm matching the data format is determined by the determination rules stored in the configuration file.

[0113] In some embodiments, the above-mentioned determination rules may be data characteristics, data distribution, outlier types, and detection requirements, etc., which are not specifically limited in the embodiments of the present application.

[0114] In some embodiments, the anomaly detection algorithm in step S420 may be a statistical-based method, a machine learning-based method, or other anomaly detection algorithms, which are not specifically limited in the embodiments of the present application.

[0115] In step S430, the data to be detected is detected by a target anomaly detection algorithm to obtain an anomaly detection result.

[0116] It can be understood that after determining the target anomaly detection algorithm corresponding to the data to be detected, the data to be detected is detected according to the target anomaly detection algorithm to determine whether there are abnormal data points in the data to be detected, and the anomaly detection result corresponding to the data to be detected is obtained. In this way, since the target anomaly detection algorithm includes at least two anomaly detection algorithms, compared with a single anomaly detection method, detecting the data to be detected by using the target anomaly detection algorithm can improve the accuracy of detection.

[0117] In some embodiments, the abnormality detection result in step S430 is usually an abnormal data point in the data to be detected.

[0118] In an embodiment of the present application, through the above-mentioned steps S410 to S430, the target anomaly detection algorithm corresponding to the data to be detected in different data formats can be automatically determined, and the target anomaly detection algorithm can be used to perform anomaly analysis on the data to be detected, thereby meeting the needs of data anomaly detection tasks with high real-time requirements, supporting anomaly detection of multi-source data, and the anomaly detection algorithm can be dynamically expanded, which is conducive to improving the detection efficiency and the accuracy of anomaly detection, so as to process the anomaly detection results in a timely manner and improve the user experience.

[0119] The SPI mechanism in step S420 is described below. Figure 5 , Figure 5 This is a schematic diagram of an implementation method of the SPI mechanism provided in the embodiment of the present application. Figure 5 As shown, developers can predefine anomaly detection interfaces that can call multiple implementors, such as Figure 5 Implementer 1 (interval determination), implementer 2 (standard deviation), implementer 3 (iForest), implementer 4 (LOF), ..., implementer n (n is a positive integer greater than 4) shown in .

[0120] here, Figure 5The caller in the code can be understood as the user who inputs the data to be detected. The implementer can also be called a service provider, which can be understood as a specific anomaly detection algorithm in essence. The anomaly detection interface detects the data to be detected by calling one or more implementers.

[0121] It should be noted that: Figure 5 It is only used for exemplary description and is not intended to limit the above. For example, implementers may also use other anomaly detection algorithms.

[0122] In some embodiments, Figure 5 The anomaly detection interface may include multiple sets of template methods, each set of template methods is used to implement a corresponding anomaly detection algorithm. For example, a set of template methods may include three steps, namely: data preprocessing, anomaly detection model training, and data anomaly prediction.

[0123] In some embodiments, Figure 5 The partial implementation code of the anomaly detection interface can be shown as follows:

[0124]

[0125] It should be noted that the above codes are only used for illustrative purposes and are not specifically limited in the embodiments of the present application.

[0126] In some embodiments, step S430 may include: step a, performing data preprocessing on the training data based on a distributed memory computing framework to obtain a data sample set; step b, sending the data sample set to each computing node of the distributed memory computing framework, and determining the target model corresponding to the target anomaly detection algorithm through the computing node; step c, detecting the data to be detected through the target model to obtain anomaly detection results.

[0127] Here, the training data can come from the data to be detected or from other data sources, as long as it can be used to train the anomaly detection model, and the embodiments of the present application do not limit this. The data preprocessing method can include data cleaning, data integration, data reduction, data transformation, feature selection, dimensionality reduction, data enhancement, and data balancing, etc., and the embodiments of the present application do not specifically limit this.

[0128] Among them, data cleaning can be used to handle missing values ​​and outliers to improve data quality; missing value processing can include deleting samples or features with missing values, or using interpolation, mean, median and other methods to fill missing values; outlier processing can be handled using statistical methods (such as Z-Score) or model-based methods. Data integration can be used to merge data from multiple data sources together to solve data redundancy and data conflict problems. Data reduction can reduce the amount of data by selecting subsets, dimensionality reduction, data compression, etc., while maintaining data integrity and availability. Data transformation can be used to convert data into a form suitable for analysis, including standardization, normalization, category coding, etc. Feature selection can select the most important features for model training, remove redundant or irrelevant features, and improve model performance and training speed. Dimensionality reduction can reduce the number of features and retain the main information through methods such as principal component analysis and linear discriminant analysis. Data enhancement can increase data diversity and improve model generalization ability by transforming data. Data balancing can be used to deal with class imbalance problems, balance class distribution through methods such as undersampling and oversampling, and improve the performance of the model on minority classes.

[0129] It is understandable that since the training data may have problems such as noise, outliers and missing values, these problems will have a negative impact on model training, so it is necessary to preprocess the training data. At the same time, considering the problem of high data calculation delay, the embodiment of the present application uses a distributed memory computing framework to preprocess the training data to obtain a preprocessed data sample set. Then, the data sample set is distributed to each computing node of the distributed memory computing framework, and the computing node trains the anomaly detection model included in the target anomaly detection algorithm through the data sample set, and the target model corresponding to the target anomaly detection algorithm is obtained. Finally, after obtaining the target model, the data to be detected is input into the target model, and the target model is used for prediction, and the output of the target model is the anomaly detection result. In this way, in the big data computing scenario, the common anomaly detection algorithm will be limited by the single-machine central processing unit (CPU), memory, etc., and the distributed memory computing framework is used to implement the above process, which can improve the efficiency of data processing, and it can also encapsulate the data uniformly, especially for data sources in various formats, the distributed memory computing framework can be used without repeated development to achieve data anomaly detection.

[0130] In some embodiments, for the above steps a to c, the implementation process of the isolation forest model is specifically introduced below, taking one of the anomaly detection algorithms included in the target anomaly detection algorithm as an improved isolation forest algorithm as an example. In order to better utilize the isolation forest algorithm to process massive data, the algorithm can be parallelized on a distributed memory computing framework (such as Spark).

[0131] See also Figure 6 , Figure 6 Schematic diagram of an implementation process of the isolation forest algorithm provided in the embodiment of the present application. Figure 6 As shown in Figure 1, the implementation process of the isolation forest algorithm includes three steps: data preprocessing, isolation forest model construction, and data anomaly detection.

[0132] Among them, data preprocessing is to use Spark to extract a batch of samples from the training data to obtain a distributed data sample set. The core of the isolation forest model is to construct an isolated binary tree, distribute the distributed data sample set to each computing node, and construct an isolated binary tree in parallel according to the calculated data set feature weights (such as Figure 6 The binary tree 1, binary tree 2, ..., binary tree m shown in the figure, where m is a positive integer greater than 2), then calculate the path length standard deviation, and merge and normalize the path length standard deviations of multiple trees to obtain a weight set. Data anomaly detection distributes the constructed feature weighted isolation forest model to each computing node, traverses each isolated binary tree in parallel for the data to be detected, calculates the path length of the data to be detected, and uses the weight set to perform weighted calculation on the path length set of each data to be detected to obtain the abnormal value of each data to be detected, thereby evaluating the abnormal situation of the data to be detected and obtaining the anomaly detection result corresponding to the data to be detected.

[0133] In some embodiments, the target model includes at least two anomaly detection models; step c may include: detecting the data to be detected by at least two anomaly detection models respectively to obtain the prediction results corresponding to each anomaly detection model; and performing weighted average calculation on the prediction results to obtain the anomaly detection results.

[0134] It is understandable that when the target model includes at least two anomaly detection models, it is necessary to consider the fusion problem of the prediction results obtained by each anomaly detection model. Based on this, after the data to be detected is detected by at least two anomaly detection models respectively and the prediction results corresponding to each anomaly detection model are obtained, different weights are set according to the performance of each anomaly detection model (such as accuracy, recall rate and other indicators), and then the prediction results are weighted averaged by weight to obtain the anomaly detection results. In this way, the anomaly detection results can be quickly obtained by the above method, and the use of at least two anomaly detection models for detection can improve the accuracy and robustness of the detection.

[0135] In some embodiments, the weights in the above weighted average calculation process can be pre-configured for different anomaly detection models to facilitate subsequent calculations.

[0136] In some embodiments, the target model includes at least two anomaly detection models; step c may include: testing the data to be tested by using the first N-1 anomaly detection models of the at least two anomaly detection models to obtain the prediction results corresponding to each anomaly detection model; using the detection results as inputs of the Nth anomaly detection model of the at least two anomaly detection models, and performing predictions by using the Nth anomaly detection model to obtain an anomaly detection result. Wherein N is a positive integer greater than 1.

[0137] It can be understood that when the target model includes at least two anomaly detection models, the prediction results of at least two anomaly detection models can be fused by stacking. The specific process can be: the first N-1 anomaly detection models in at least two anomaly detection models are used to detect the data to be detected, and the prediction results corresponding to each anomaly detection model are obtained. Then, the detection results are used as the input of the Nth anomaly detection model, and the input is predicted by the Nth anomaly detection model to obtain the anomaly detection result. In this way, the complementarity between different anomaly detection models can be captured through the above method, thereby improving the accuracy and robustness of detection.

[0138] In some embodiments, step a may include: performing data preprocessing on the training data according to the data type through a distributed memory computing framework to obtain processed training data; converting the processed training data into a data format supported by the distributed memory computing framework to obtain a data sample set.

[0139] Here, the types of data may include: text data, category data, and time series data, etc., which are not limited in the embodiments of the present application.

[0140] It is understandable that since the training data may contain noise, outliers, and missing values, these problems will have an adverse effect on data anomaly detection. Therefore, it is necessary to preprocess the training data. At the same time, considering the problem of high data calculation latency, the embodiment of the present application uses a distributed memory computing framework to preprocess the training data according to the type of data to obtain processed training data. Since different distributed memory computing frameworks support different data formats, it is also necessary to convert the processed training data into a data format supported by the distributed memory computing framework to obtain a data sample set. In this way, the distributed memory computing framework can perform parallel calculations on the data sample set, improve processing speed and processing efficiency, and save time for data anomaly detection.

[0141] In some embodiments, step S420 may include: determining at least two anomaly detection algorithms corresponding to the data to be detected through a configuration file of the SPI mechanism according to the data format of the data to be detected, wherein the configuration file includes a mapping relationship between different data formats and anomaly detection algorithms; and determining a target anomaly detection algorithm based on the at least two anomaly detection algorithms.

[0142] It is understandable that developers can determine the mapping relationship between different data formats and anomaly detection algorithms in advance through the configuration file of the SPI mechanism. In this way, after obtaining the data to be detected, at least two anomaly detection algorithms corresponding to the data to be detected can be determined through the configuration file according to the data format of the data to be detected. Then, based on at least two anomaly detection algorithms, at least two anomaly detection algorithms can be superimposed and used, or a new anomaly detection algorithm can be constructed based on at least two anomaly detection algorithms to obtain a target anomaly detection algorithm. The target anomaly detection algorithm can be at least two anomaly detection algorithms, or a new anomaly detection algorithm constructed based on at least two anomaly detection algorithms. The embodiments of the present application do not limit this. In this way, the accuracy and timeliness of detection can be improved by integrating multiple anomaly detection algorithms.

[0143] In some embodiments, after step S430, the data anomaly detection method may further include: obtaining audit rules and audit indicators corresponding to the data to be detected; analyzing the anomaly detection results through the audit rules and audit indicators to obtain a data quality report corresponding to the data to be detected.

[0144] Here, the audit rules and audit indicators can be determined according to the data format and data characteristics corresponding to the data to be detected, and the audit rules and audit indicators corresponding to different data to be detected may be different. Audit rules may include: integrity rules, uniqueness rules, consistency rules, legality rules, accuracy rules, logic rules, outlier rules, completeness rules, traceability rules, and data security rules, etc., which are not specifically limited in the embodiments of the present application. Audit indicators may include: accuracy, null value rate, standard deviation, data accessibility rate, robustness, etc., which are not specifically limited in the embodiments of the present application.

[0145] It can be understood that after obtaining the anomaly detection results, the audit rules and audit indicators corresponding to the data to be detected are obtained, and the anomaly detection results are analyzed through the audit rules and audit indicators to obtain the data quality report corresponding to the data to be detected. The data quality report may include data distribution, abnormal data points, data accuracy, etc. In this way, the data quality report can more intuitively reflect the data anomaly detection results, which is conducive to further decision-making analysis based on the data quality report.

[0146] Figure 7FIG. 1 is another implementation flow diagram of the data anomaly detection method provided in the embodiment of the present application. Figure 7 As shown, the embodiment of the present application mainly introduces the process of obtaining anomaly detection results, which may specifically include steps S710 to S750.

[0147] In step S710, data to be detected is obtained.

[0148] In step S720, based on the SPI mechanism, a target anomaly detection algorithm corresponding to the data to be detected is determined.

[0149] In step S730, based on the distributed memory computing framework, data preprocessing is performed on the training data to obtain a data sample set.

[0150] In step S740, the data sample set is sent to each computing node of the distributed memory computing framework, and the target model corresponding to the target anomaly detection algorithm is determined through the computing nodes.

[0151] In step S750, the data to be detected is detected by using the target model to obtain an abnormality detection result.

[0152] Based on the same inventive concept, the present application also provides a data anomaly detection device, such as the data anomaly detection device 155 in the above embodiment. Figure 2 As shown, the data anomaly detection device 155 includes: a first acquisition module 1551, used to acquire the data to be detected, wherein the data to be detected includes at least one data format; a determination module 1552, based on the SPI mechanism, determines the target anomaly detection algorithm corresponding to the data to be detected, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms; a detection module 1553, detects the data to be detected by using the target anomaly detection algorithm to obtain an anomaly detection result.

[0153] In some embodiments, the detection module 1553 includes: a first determination unit, used to perform data preprocessing on the training data based on a distributed memory computing framework to obtain a data sample set; a second determination unit, used to send the data sample set to each computing node of the distributed memory computing framework, and determine the target model corresponding to the target anomaly detection algorithm through the computing node; a detection unit, used to detect the data to be detected through the target model to obtain anomaly detection results.

[0154] In some embodiments, the target model includes at least two anomaly detection models; the detection unit is used to perform the following steps: respectively detecting the data to be detected by at least two anomaly detection models to obtain the prediction results corresponding to each anomaly detection model; performing weighted average calculation on the prediction results to obtain the anomaly detection results.

[0155] In some embodiments, the target model includes at least two anomaly detection models; a detection unit is used to perform the following steps: detecting the data to be detected by using the first N-1 anomaly detection models of the at least two anomaly detection models to obtain a prediction result corresponding to each anomaly detection model, wherein N is a positive integer greater than 1; using the detection result as an input of the Nth anomaly detection model of the at least two anomaly detection models, and performing prediction by the Nth anomaly detection model to obtain an anomaly detection result.

[0156] In some embodiments, the first determination unit is used to perform the following steps: preprocess the training data according to the data type through a distributed memory computing framework to obtain processed training data; convert the processed training data into a data format supported by the distributed memory computing framework to obtain a data sample set.

[0157] In some embodiments, the determination module 1552 is used to perform the following steps: determine at least two anomaly detection algorithms corresponding to the data to be detected through a configuration file of the SPI mechanism according to the data format of the data to be detected, wherein the configuration file includes a mapping relationship between different data formats and anomaly detection algorithms; and determine a target anomaly detection algorithm based on at least two anomaly detection algorithms.

[0158] In some embodiments, the above-mentioned device also includes: a second acquisition module, which is used to acquire the audit rules and audit indicators corresponding to the data to be detected after the data to be detected is detected by using the target anomaly detection algorithm and the anomaly detection result is obtained; an analysis module, which is used to analyze the anomaly detection result through the audit rules and audit indicators to obtain a data quality report corresponding to the data to be detected.

[0159] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0160] It should be noted that, in the embodiment of the present application, if the above-mentioned data anomaly detection method is implemented in the form of a software function module, and when it is sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0161] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0162] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0163] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes some or all of the steps for implementing the above method.

[0164] The present application embodiment provides a computer program product, including a computer program or an instruction, which implements some or all of the steps in the above method when the computer program or instruction is executed by a processor. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0165] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0166] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0167] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0170] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0171] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0172] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0173] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data anomaly detection method, characterized in that: include: Acquire data to be detected, wherein the data to be detected includes at least one data format; Based on the service provider interface SPI mechanism, determine a target anomaly detection algorithm corresponding to the data to be detected, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms; The target anomaly detection algorithm is used to detect the data to be detected to obtain an anomaly detection result.

2. The method according to claim 1, characterized in that The detecting the data to be detected by using the target anomaly detection algorithm to obtain an anomaly detection result includes: Based on the distributed memory computing framework, the training data is preprocessed to obtain a data sample set; Sending the data sample set to each computing node of the distributed memory computing framework, and determining the target model corresponding to the target anomaly detection algorithm through the computing node; The data to be detected is detected by using the target model to obtain the anomaly detection result.

3. The method according to claim 2, characterized in that The target model includes at least two anomaly detection models; The detecting the data to be detected by using the target model to obtain the abnormality detection result includes: Using the at least two anomaly detection models to detect the data to be detected respectively, and obtaining a prediction result corresponding to each anomaly detection model; A weighted average calculation is performed on the prediction results to obtain the anomaly detection result.

4. The method according to claim 2, characterized in that: The target model includes at least two anomaly detection models; The detecting the data to be detected by using the target model to obtain the abnormality detection result includes: Detecting the data to be detected by using the first N-1 anomaly detection models of the at least two anomaly detection models to obtain a prediction result corresponding to each anomaly detection model, wherein N is a positive integer greater than 1; The detection result is used as an input of an Nth anomaly detection model among the at least two anomaly detection models, and prediction is performed through the Nth anomaly detection model to obtain the anomaly detection result.

5. The method according to claim 2, characterized in that: The distributed memory computing framework is used to preprocess the training data to obtain a data sample set, including: By using the distributed memory computing framework, the training data is preprocessed according to the data type to obtain processed training data; The processed training data is converted into a data format supported by the distributed memory computing framework to obtain the data sample set.

6. The method according to claim 1, characterized in that The determining, based on the SPI mechanism, a target anomaly detection algorithm corresponding to the data to be detected includes: According to the data format of the data to be detected, at least two anomaly detection algorithms corresponding to the data to be detected are determined through a configuration file of the SPI mechanism, wherein the configuration file includes a mapping relationship between different data formats and anomaly detection algorithms; Based on the at least two anomaly detection algorithms, the target anomaly detection algorithm is determined.

7. The method according to any one of claims 1 to 6, characterized in that: After detecting the data to be detected by the target anomaly detection algorithm to obtain an anomaly detection result, the method further includes: Obtaining the audit rules and audit indicators corresponding to the data to be tested; The abnormal detection result is analyzed by the audit rules and the audit indicators to obtain a data quality report corresponding to the data to be detected.

8. A data anomaly detection device, characterized in that: include: A first acquisition module, used to acquire data to be detected, wherein the data to be detected includes at least one data format; A determination module, based on a service provider interface SPI mechanism, determines a target anomaly detection algorithm corresponding to the data to be detected, wherein the target anomaly detection algorithm includes at least two anomaly detection algorithms; The detection module detects the data to be detected by using the target anomaly detection algorithm to obtain an anomaly detection result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images