Data anomaly detection method and device, processor and electronic equipment

Abstract

The application discloses a data anomaly detection method and device, a processor and electronic equipment. The method comprises the following steps: obtaining the identification information of at least one preset calculation module and at least one preset storage module; filling the identification information into a preset adaptation table of a preset data warehouse; predicting the adaptation probability of the preset calculation module and the preset storage module according to a preset adaptation model, and filling the adaptation probability into the preset adaptation table, wherein the preset adaptation model is obtained by training a deep learning algorithm based on a self-attention mechanism; performing data caching of the preset adaptation table by calling stream data and a remote dictionary service through the preset calculation module, to obtain a data stream processing table, wherein the remote dictionary service and the preset calculation module establish a mapping relationship in the form of a key-value pair; and analyzing the data stream processing table to realize data anomaly detection. The application solves the technical problem of low data anomaly detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and in particular, to a data anomaly detection method and device, a processor and an electronic device. BACKGROUND

[0002] With the continuous development of the field of big data, more and more concepts are proposed and applied to production, and the data lake concept is one of them. The data lake is a centralized storage repository that can store structured and unstructured data of any size, supporting big data and AI computing. Data Lake Formation (DLF) as a core component of the cloud-native data lake architecture helps users quickly build cloud-native data lake solutions. Data lake construction provides unified metadata management, enterprise-level permission control, and seamless integration with multiple computing engines, breaking down data silos and uncovering business value.

[0003] However, when using the data lake to build the adaptive relationship between the data storage and the computing engine, the data processing process is relatively complex, and if data anomalies occur, the data anomaly detection cannot be quickly realized.

[0004] At present, there is no effective solution to the above-mentioned problem of low data anomaly detection efficiency. SUMMARY

[0005] The embodiments of the present application provide a data anomaly detection method, device, processor and electronic device to at least solve the technical problem of low data anomaly detection efficiency.

[0006] According to an aspect of an embodiment of the present application, a data anomaly detection method is provided, comprising: obtaining the identification information of at least one preset computing module and at least one preset storage module; filling the identification information into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for the adaptation of the preset computing module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship of the preset computing module and the preset storage module; predicting the adaptation probability of the preset computing module and the preset storage module according to a preset adaptation model, and filling the adaptation probability into the preset adaptation table, wherein the preset adaptation model is trained based on a deep learning algorithm of a self-attention mechanism; calling the stream data and a preset front end through the preset computing module to perform data caching of the preset adaptation table, to obtain a data stream processing table, wherein the preset front end and the preset computing module establish a mapping relationship in the form of key-value pairs; and analyzing the data stream processing table to realize data anomaly detection.

[0007] Optionally, before obtaining the identification information of the at least one preset computing module and the at least one preset storage module, the method further comprises: obtaining a preset processing architecture of data, wherein the preset processing architecture at least comprises: a computing layer and a storage layer, the computing layer comprises at least one preset computing module, and the storage layer comprises at least one preset storage module; and adding the identification information to the at least one preset computing module and the at least one preset storage module through the preset data warehouse.

[0008] Optionally, before predicting the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model, the method further comprises: obtaining historical flow data of the at least one preset computing module and the at least one preset storage module executed in a preset time period; identifying an abnormal feature of the historical flow data; and training the preset adaptation model based on the abnormal feature of the historical flow data.

[0009] Optionally, training the preset adaptation model based on the abnormal feature of the historical flow data comprises: identifying an abnormal keyword in the historical flow data; extracting a time feature of the abnormal keyword in a time dimension; extracting a space feature of the abnormal keyword in a space dimension; and determining the preset adaptation model based on the time feature and the space feature of the abnormal keyword.

[0010] Optionally, before performing data caching of the preset adaptation table through the preset computing module and a preset front end, the method further comprises: converting an execution code in the preset computing module into a logical execution plan; converting the logical execution plan into a physical execution plan through an optimizer; and generating an executable calling function through a code generation technology, wherein the calling function is used to configure a calling plan of the preset computing module.

[0011] According to another aspect of the embodiments of the present application, a data anomaly detection apparatus is also provided, comprising: a first obtaining unit, configured to obtain identification information of at least one preset computing module and at least one preset storage module; a storage unit, configured to fill the identification information into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is configured to provide a running environment for adaptation of the preset computing module and the preset storage module, and the preset adaptation table is configured to record an adaptation relationship between the preset computing module and the preset storage module; a prediction unit, configured to predict an adaptation probability of the preset computing module and the preset storage module according to a preset adaptation model, and fill the adaptation probability into the preset adaptation table, wherein the preset adaptation model is trained based on a deep learning algorithm of a self-attention mechanism; a calling unit, configured to call stream data and a preset front end through the preset computing module to perform data caching of the preset adaptation table, and obtain a data stream processing table, wherein the preset front end and the preset computing module establish a mapping relationship in the form of a key-value pair; and an analysis unit, configured to analyze the data stream processing table to implement data anomaly detection.

[0012] Optionally, the apparatus further comprises: a second obtaining unit, configured to obtain a preset processing architecture of data before obtaining the identification information of the at least one preset computing module and the at least one preset storage module, wherein the preset processing architecture at least comprises a computing layer and a storage layer, the computing layer comprises the at least one preset computing module, and the storage layer comprises the at least one preset storage module; and an adding unit, configured to add the identification information for the at least one preset computing module and the at least one preset storage module through the preset data warehouse.

[0013] Optionally, the apparatus further comprises: a third obtaining unit, configured to obtain historical stream data executed by the at least one preset computing module and the at least one preset storage module in a preset time period before predicting the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model; a first identifying unit, configured to identify an abnormal feature of the historical stream data; and a training unit, configured to train the preset adaptation model based on the abnormal feature of the historical stream data.

[0014] According to another aspect of the embodiments of the present application, a processor is also provided, configured to run a program, wherein the program performs the above data anomaly detection method when running.

[0015] According to another aspect of the embodiments of the present application, an electronic device is also provided, characterized by comprising one or more processors and a memory, wherein the memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above data anomaly detection method.

[0016] In the embodiment of the present application, the identification information of at least one preset computing module and at least one preset storage module is obtained; the identification information is filled into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for the adaptation of the preset computing module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship of the preset computing module and the preset storage module; the adaptation probability of the preset computing module and the preset storage module is predicted according to a preset adaptation model, and the adaptation probability is filled into the preset adaptation table, wherein the preset adaptation model is obtained by training a deep learning algorithm based on a self-attention mechanism; the data cache of the preset adaptation table is performed through the preset computing module calling the stream data and the preset front end, and a data stream processing table is obtained, wherein the preset front end and the preset computing module establish a mapping relationship in the form of a key-value pair; the data stream processing table is analyzed to realize data anomaly detection, and since the data stream processing table records the processing process of the data stream, the detection of the data running health condition and the fault positioning can be quickly performed by analyzing the data stream processing table, thereby realizing the technical effect of quickly performing data anomaly detection, and further solving the technical problem of low data anomaly detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0018] Figure 1 is a flow chart of a data anomaly detection method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a flow optimization method of a data lake combined with a Flink data engine and integrated with redis according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a work flow specific description of a Flink SQL engine according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a data anomaly detection device according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0024] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to include all the steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present application, a method embodiment of data anomaly detection is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0026] Figure 1 is a flowchart of a data anomaly detection method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0027] Step S102, obtaining the identification information of at least one preset computing module and at least one preset storage module;

[0028] Step S104, filling the identification information into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for the adaptation of the preset computing module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship between the preset computing module and the preset storage module;

[0029] Step S106, predicting the adaptation probability of the preset computing module and the preset storage module according to a preset adaptation model, and filling the adaptation probability into the preset adaptation table, wherein the preset adaptation model is trained based on a deep learning algorithm of self-attention mechanism;

[0030] In step S108, the data stream and the preset front end are called by the pre-design calculation module to perform data caching of the preset adaptation table to obtain a data stream processing table, wherein the preset front end and the pre-design calculation module establish a mapping relationship in the form of a key-value pair.

[0031] In step S110, the data stream processing table is analyzed to implement data anomaly detection.

[0032] In the embodiment of the present application, the identification information of at least one pre-design calculation module and at least one preset storage module is obtained; the identification information is filled into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for the adaptation of the pre-design calculation module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship of the pre-design calculation module and the preset storage module; the adaptation probability of the pre-design calculation module and the preset storage module is predicted according to a preset adaptation model, and the adaptation probability is filled into the preset adaptation table, wherein the preset adaptation model is obtained by training a deep learning algorithm based on a self-attention mechanism; the data stream and the preset front end are called by the pre-design calculation module to perform data caching of the preset adaptation table to obtain a data stream processing table, wherein the preset front end and the pre-design calculation module establish a mapping relationship in the form of a key-value pair; the data stream processing table is analyzed to implement data anomaly detection, and since the data stream processing table records the processing process of the data stream, the data running health condition can be quickly detected and fault positioning can be performed by analyzing the data stream processing table, thereby achieving the technical effect of quickly performing data anomaly detection, and further solving the technical problem of low data anomaly detection efficiency.

[0033] In the above step S102, the pre-design calculation module includes a Flink module, a spark module, a Hive module, and a Presto module.

[0034] In the above step S102, the preset storage module includes an ORC module, a Parquet module, and an AVRO module.

[0035] In the above step S104, the preset data warehouse is a real-time data warehouse based on an iceberg data lake technology deployed in a central server.

[0036] In the above step S104, the identification information is filled into a preset adaptation table of a preset data warehouse, and the format can be: "Radis###Spark###Flink###Hive###Presto###ORC###Parquet###AVRO", wherein Radis can be a front end, and is mapped with the Flink module.

[0037] In the step S106, the adaptation probability is filled in the preset adaptation table, and the format can be: "Radis_35%###Spark_35%###Flink_35%###Hive_35%###Presto_35%###ORC_35%###Parquet_35%###AVRO_35%", wherein 35% is the adaptation probability between the modules.

[0038] In the step S108, the data cache of the preset adaptation table is obtained by calling the stream data and the preset front end through the preset computing module, and the format of the data stream processing table can be: "Radis_35%###Spark_35%###Flink_35_LogicalPlan34%_Optimizer34%_Physical Plan34%_TransFormation34%_JobGraph34%###Hive_35%###Presto_35%###ORC_35%###Parquet_35%###AVRO_35%", wherein the key-value storage form of redis is used to connect with Flink to form Flink-connection-redis. Through the establishment of the key and value mapping relationship, a session table is formed for FlinkSQL calling.

[0039] As an optional embodiment, before the identification information of the at least one preset computing module and the at least one preset storage module is obtained, the method further comprises: obtaining a preset processing architecture of data, wherein the preset processing architecture at least comprises: a computing layer and a storage layer, the computing layer comprises at least one preset computing module, and the storage layer comprises at least one preset storage module; and adding identification information for the at least one preset computing module and the at least one preset storage module through a preset data warehouse.

[0040] Optionally, the preset data warehouse is located between the computing layer and the storage layer, and is used to provide a basic environment for performance adaptation optimization of big data operation.

[0041] In the above embodiment of the application, the real-time data warehouse based on the iceberg data lake technology is deployed in the central server, the identification information of each module of the computing layer and the storage layer is provided based on the iceberg data lake technology, and then the cooperation relationship between the preset computing module and the preset storage module can be recorded by recording the identification information of each module in the preset adaptation table.

[0042] As an optional embodiment, before predicting the adaptation probability of the pre-designed computing module and the pre-set storage module according to the pre-set adaptation model, the method further comprises: acquiring historical flow data executed by at least one pre-designed computing module and at least one pre-set storage module in a pre-set time period; identifying an abnormal feature of the historical flow data; and training the pre-set adaptation model based on the abnormal feature of the historical flow data.

[0043] The above embodiment of the present application continuously analyzes the flow data in a period of time based on the deep learning algorithm Transformer of the self-attention mechanism, discovers the potential features before the abnormal keyword factors of the flow data for the data sequence, and makes the flow data warning more accurate.

[0044] It should be noted that the historical flow data is divided into the following types of data according to the Iceberg (Table Format) characteristics:

[0045] 1. Support transaction (ACID), upstream data writing is visible, does not affect the current data processing task, simplifies ETL; provides upsert and merge into capabilities, which can greatly reduce data warehouse delay.

[0046] Among them, the abnormal keyword is that the data in the warehouse is not synchronized with the upstream writing.

[0047] 2. Extensible metadata, snapshot isolation, and all modifications to the file list are atomic operations.

[0048] Among them, the abnormal keyword: the thread running phase is switched to another thread, or the thread is terminated in the middle.

[0049] It should be noted that the atomic operation refers to an operation that cannot be interrupted by the thread scheduling mechanism; once this operation starts, it runs to the end without any context switch (switching to another thread) in the middle.

[0050] 3. Supports stream batch processing and supports multiple storage formats and flexible file organization: provides a stream-based incremental computing model and a batch-based full table computing model. Batch processing and stream tasks can use the same storage model, and data is no longer isolated; Iceberg supports hidden partitioning and partition evolution, which facilitates business data partitioning strategy updates. Supports Parquet, Avro, and ORC storage formats.

[0051] Among them, the abnormal keyword: data storage failure or partial loss, Parquet, Avro, and ORC storage format exception.

[0052] As an optional embodiment, the preset adaptive model is trained based on the abnormal features of the historical flow data, including: identifying an abnormal keyword in the historical flow data; extracting a time feature of the abnormal keyword in a time dimension; extracting a space feature of the abnormal keyword in a space dimension; and determining the preset adaptive model based on the time feature and the space feature of the abnormal keyword.

[0053] Optionally, for the time dimension, positional encoding can be used for time sequence encoding, and attention can be used to explore the time feature of the time dimension.

[0054] Optionally, for the space dimension, multi-head attention mechanism can be used to extract the space features of different space dimensions, so that the feature extraction is more sufficient.

[0055] As an optional embodiment, before the data cache of the preset adaptive table is performed by calling the flow data and the preset front end through the pre-design calculation module, the method further includes: converting the execution code in the pre-design calculation module into a logical execution plan; converting the logical execution plan into a physical execution plan through an optimizer; and generating an executable calling function through a code generation technology, wherein the calling function is used to configure the calling plan of the pre-design calculation module.

[0056] Optionally, the calling function JobGraph is used to optimize the flow StreamGraph, such as setting which operators can be chained to reduce network overhead.

[0057] In the above embodiments of the present application, some operators in the graph structure of the Flink task in the pre-design calculation module are chained together, which can reduce serialization and network overhead and improve efficiency.

[0058] The application also provides a preferred embodiment, which provides a data lake combined with a Flink data engine and integrated with a redis use process optimization method.

[0059] The method of the application puts forward higher requirements for big data involved in operation and maintenance complex scenarios, that is, based on the iceberg data lake technology, a real-time data warehouse is established. Flink is used as the main computing engine, and Flink is a distributed processing engine for stream data and batch data. At the same time, the redis framework is used for table data caching. And using the redis key-value storage form, combined with FlinkSQL to complete data statistics. To solve the key link of the data lake solution, that is, the adaptation problem between data storage and computing engine, by using the artificial intelligence to build an adaptive model with the support of the Exactly-once feature of Flink's stateful computing, an optimal balance point between the performance and resource overhead of big data in the computing layer and the corresponding consumption of storage and infrastructure and performance overhead is provided. With the continuous increase of data volume and the strict requirements of business on timeliness, the role of real-time data warehouse becomes more and more important. Iceberg, with the characteristics of ACID transaction, time travel and excellent abstraction, as well as wide support for computing engines such as Spark and Flink, as the core component of real-time data warehouse, can shorten the import process, facilitate data changes and speed up data reading.

[0060] Figure 2 It is a schematic diagram of a data lake combined with a Flink data engine and integrated with a redis use process optimization method according to an embodiment of the application, as shown in Figure 2 The steps are as follows:

[0061] Step one, first, deploy a real-time data warehouse based on the iceberg data lake technology on the central server. Formally, iceberg is a table format. That is, an intermediate layer based on the computing layer (Flink, spark, Hive, Presto) and the storage layer (ORC, Parquet, AVRO). Provide a basic environment for performance adaptation optimization of big data operation. And provide a unique identifier for each module on the level: Flag: modules are separated by ###.

[0062] Flag format of step one:

[0063] "Radis###Spark###Flink###Hive###Presto###ORC###Parquet###AVRO"

[0064] Step two, according to the Iceberg(Table Format) feature classification aggregate data and using a deep learning algorithm based on self-attention mechanism of the algorithm Transformer algorithm combined with the construction of the adaptive model for operation, according to the results of the operation on the Iceberg(Table Format) upper and lower layers of each module data adaptive scheme probability prediction. And update to the unique identification Flag underscore interval.

[0065] It should be noted that, since the Flink SQL engine principle using code generation technology(CodeGen) based on self-attention mechanism of the deep learning algorithm Transformer algorithm for code compilation into an executable JobGraph principle, therefore using based on self-attention mechanism of the deep learning algorithm Transformer algorithm, to build adaptive model.

[0066] Step two of the Flag format:

[0067] "Radis_35%###Spark_35%###Flink_35%###Hive_35%###Presto_35%###ORC_35%###Parquet_35%###AVRO_35%"。

[0068] Optionally, according to the access history database according to the Iceberg(Table Format) feature into three types of data.

[0069] 1, support transaction(ACID), upstream data write can be seen, does not affect the current data processing task, simplify ETL; provide upsert and merge into ability, can greatly reduce the data into the warehouse delay;

[0070] Abnormal keywords: data into the warehouse and upstream write out of sync.

[0071] 2, scalable metadata, snapshot isolation and all modifications to the file list are atomic operations.

[0072] Abnormal keywords: thread running phase switch to another thread, or the thread is terminated.

[0073] It should be noted that the atomic operation refers to the operation that will not be interrupted by the thread scheduling mechanism; such operation once started, always run to the end, there will be no context switch(in another thread) in the middle.

[0074] 3. Support stream batch processing, support multiple storage formats and flexible file organization: Provide incremental computing model based on streaming and full table computing model based on batch processing. Batch and stream tasks can use the same storage model, and data is no longer isolated; Iceberg supports hidden partitioning and partition evolution, making it easy for businesses to update data partitioning strategies. Supports Parquet, Avro, and ORC storage formats.

[0075] Abnormal keywords: data storage failure or partial loss, Parquet, Avro, and ORC storage format exceptions.

[0076] Optionally, the specific description of the adaptive model is:

[0077] The deep learning algorithm Transformer based on self-attention mechanism continuously analyzes a period of stream data, and finds the potential features before the abnormal keyword factors of the stream data sequence, making the stream data warning more accurate.

[0078] First, access the stream abnormal keyword factor feature data in the historical database.

[0079] Second, to adapt to the model processing, the non-uniformly sampled data of different nodes is processed by difference. The quadratic difference method is used to interpolate every 3 adjacent points to obtain quadratic interpolation. That is, the abnormal keyword factor data optimized by artificial intelligence algorithm. The advantages of this method are:

[0080] 1. Uniform interval, more matching with the time series processing of the deep learning algorithm Transformer based on self-attention mechanism.

[0081] 2. More realistic restoration of missing data in network traffic scenarios.

[0082] The quadratic difference formula is as follows:

[0083]

[0084] x = current value of the classification object

[0085] y = 3 adjacent points of the classification object.

[0086] i = order number of the classification object

[0087] Finally, the stream data optimized by artificial intelligence algorithm is combined with the adaptive model, and the continuous time series data is used for training to find the correlation of different time and space dimensions, so as to realize accurate stream data abnormal keyword factor warning. Complete the process of Transformer warning model training based on abnormal keyword factors based on self-attention mechanism.

[0088] Model formula:

[0089] S1. For the time dimension, use positional encoding for time encoding, and use attention to explore the time characteristics of the time dimension.

[0090] Optionally, the positional encoding PositionEncoding = cos2(pos / N), wherein N is the adjustable length size.

[0091] Optionally, the attention output Attention_output = Attention(Q, K, V), wherein Q is the key-value mapping of the query feature corresponding to Redis, K is the key-value mapping of the feature to be matched corresponding to Redis, and V is the key-value mapping of the monitoring stream data corresponding to Redis.

[0092] S2. For the spatial dimension, use multi-head attention mechanism to extract different multi-space dimension features, so that the feature extraction is more sufficient.

[0093] Optionally, Headi = Attention(Qi, Ki, Vi); MultiHead(Q, K, V) = Concat(head1,..., headh)*WO, wherein Headi is the result obtained by time attention, and multi-space dimension features are fused through MultiHead.

[0094] Step three, Iceberg(Table Format) transmits the data in unified format to Flink, and Flink calls the stream data and front-end Redis through FlinkSQL to realize Iceberg(Table Format) data caching. And update to the unique identification: Flag.

[0095] Optionally, before JobGraph is generated, the deep learning algorithm Transformer based on self-attention mechanism is used to adapt the model to predict whether the FlinkSQL between Flink-Redis is abnormal. So as to ensure the normal work of Flink engine. Finally, through program analysis of Flag identification, the data lake Iceberg(Table Format) combined with Flink data engine and integrated Redis can be analyzed to obtain the whole process of data stream processing and running health condition and timely track fault location.

[0096] The format of Flag in step three is:

[0097] "Radis_35%###Spark_35%###Flink_35_LogicalPlan34%_Optimizer34%_Physical Plan34%_TransFormation34%_JobGraph34%%###Hive_35%###Presto_35%###ORC_35%###Parquet_35%###AVRO_35%"

[0098] Optionally, the key-value storage form of redis is used to connect with Flink to form Flink-connection-redis. By establishing a key, value mapping relationship, a session table is formed. For FlinkSQL to call.

[0099] Figure 3 is a schematic diagram of a work flow of a Flink SQL engine according to an embodiment of the application. As shown in Figure 3 The SQL / Table API has the following steps from input to compilation into an executable JobGraph:

[0100] S31, convert the SQL text / Table API code into a logical execution plan (Logical Plan).

[0101] S32, the logical execution plan is converted into a physical execution plan (Physical Plan) by an optimizer.

[0102] S33, generate a deep learning algorithm Transformation based on a self-attention mechanism through code generation technology and further compile into an executable JobGraph for submission and running.

[0103] It should be noted that the JobGraph is an optimization of the StreamGraph, such as setting which operators can be chained to reduce network overhead. In the graph structure of the Flink task, some operators are chained together (to reduce serialization and network overhead and improve efficiency).

[0104] Optionally, the specific process of SQL implementation includes:

[0105] S41, convert the SQL text / Table API code into a logical execution plan.

[0106] S42, the SQL / Table API converts the SQL parsing into an AST abstract syntax tree through the calcite framework.

[0107] S43. The SQL Validator retrieves metadata from the Catalog to validate expressions, table information, etc., and converts them into relational algebra expressions (RelNode).

[0108] S44. The optimizer then transforms the relational algebra expression into a logical execution plan for the initial state.

[0109] According to an embodiment of the present invention, a device embodiment for data anomaly detection is also provided. It should be noted that the data anomaly detection device can be used to execute the data anomaly detection method in the embodiments of the present invention, and the data anomaly detection method in the embodiments of the present invention can be executed in the data anomaly detection device.

[0110] Figure 4 This is a schematic diagram of a data anomaly detection device according to an embodiment of the present invention, such as... Figure 4 As shown, the device may include: a first acquisition unit 41, used to acquire identification information of at least one preset computing module and at least one preset storage module; a storage unit 43, used to fill the identification information into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide an operating environment for the adaptation of the preset computing module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship between the preset computing module and the preset storage module; a prediction unit 45, used to predict the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model, and fill the adaptation probability into the preset adaptation table, wherein the preset adaptation model is trained based on a deep learning algorithm with a self-attention mechanism; a calling unit 47, used to call streaming data and a preset front-end through the preset computing module to cache the data in the preset adaptation table, thereby obtaining a data stream processing table, wherein the preset front-end and the preset computing module establish a mapping relationship in the form of key-value pairs; and an analysis unit 49, used to analyze the data stream processing table to realize data anomaly detection.

[0111] It should be noted that the first acquisition unit 41 in this embodiment can be used to execute step S102 in this application embodiment, the storage unit 43 in this embodiment can be used to execute step S104 in this application embodiment, the prediction unit 45 in this embodiment can be used to execute step S106 in this application embodiment, the first calling unit 47 in this embodiment can be used to execute step S108 in this application embodiment, and the analysis unit 49 in this embodiment can be used to execute step S110 in this application embodiment. The examples and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0112] In this embodiment of the invention, identification information of at least one preset computing module and at least one preset storage module is obtained; the identification information is filled into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for the adaptation of the preset computing module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship between the preset computing module and the preset storage module; the adaptation probability of the preset computing module and the preset storage module is predicted according to the preset adaptation model, and the adaptation probability is filled into the preset adaptation table, wherein the preset adaptation model is trained based on a deep learning algorithm with a self-attention mechanism; the preset computing module calls streaming data and a preset front-end to cache the data in the preset adaptation table to obtain a data stream processing table, wherein the preset front-end and the preset computing module establish a mapping relationship in the form of key-value pairs; the data stream processing table is analyzed to realize data anomaly detection. Since the data stream processing table records the data stream processing process, the health status of data operation and fault location can be quickly detected and located by analyzing the data stream processing table, thereby achieving the technical effect of rapid data anomaly detection and solving the technical problem of low efficiency in data anomaly detection.

[0113] As an optional embodiment, the apparatus further includes: a second acquisition unit, configured to acquire a preset processing architecture of data before acquiring the identification information of at least one preset computing module and at least one preset storage module, wherein the preset processing architecture includes at least one computing layer and one storage layer, the computing layer including at least one preset computing module and the storage layer including at least one preset storage module; and an addition unit, configured to add identification information to at least one preset computing module and at least one preset storage module through a preset data warehouse.

[0114] As an optional embodiment, the apparatus further includes: a third acquisition unit, configured to acquire historical stream data of at least one preset computing module and at least one preset storage module executed within a preset time period before predicting the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model; a first identification unit, configured to identify abnormal features of the historical stream data; and a training unit, configured to train the preset adaptation model based on the abnormal features of the historical stream data.

[0115] As an optional embodiment, the training unit includes: a second identification unit for identifying abnormal keywords in historical stream data; a first extraction unit for extracting the temporal features of the abnormal keywords in the time dimension; a second extraction unit for extracting the spatial features of the abnormal keywords in the spatial dimension; and a determination unit for determining a preset adaptation model based on the temporal and spatial features of the abnormal keywords.

[0116] As an optional embodiment, the device further comprises: a first conversion unit, configured to convert the execution code SQL / TableAPI in the pre-design calculation module into a logical execution plan before the data buffer of the preset adaptation table is called by the pre-design calculation module and the preset front end; a second conversion unit, configured to convert the logical execution plan into a physical execution plan by an optimizer; and a generation unit, configured to generate an executable calling function JobGraph by a code generation technique, wherein the calling function JobGraph is used to configure the calling plan of the pre-design calculation module.

[0117] The data anomaly detection device comprises a processor and a memory, all of the units are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0118] The processor comprises a core, and the core calls the corresponding program units in the memory.

[0119] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (Flash RAM), and the memory comprises at least one memory chip.

[0120] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the data anomaly detection method.

[0121] The embodiment of the present application provides a processor, which is used for running a program, and the program is executed to realize the data anomaly detection method.

[0122] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application, as Figure 5As shown, the embodiment of the present application provides an electronic device 50, the device comprising a processor 52, a memory 54, and a program stored on the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining identification information of at least one preset computing module and at least one preset storage module; filling the identification information into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for the adaptation of the preset computing module and the preset storage module, and the preset adaptation table is used to record the adaptation relationship between the preset computing module and the preset storage module; predicting an adaptation probability of the preset computing module and the preset storage module according to a preset adaptation model, and filling the adaptation probability into the preset adaptation table, wherein the preset adaptation model is obtained by training a deep learning algorithm based on a self-attention mechanism; performing data caching of the preset adaptation table by calling stream data and a preset front end through the preset computing module, to obtain a data stream processing table, wherein the preset front end and the preset computing module establish a mapping relationship in the form of a key-value pair; and analyzing the data stream processing table to implement data anomaly detection.

[0123] Optionally, the processor implements the following steps when executing the program: before obtaining the identification information of the at least one preset computing module and the at least one preset storage module, obtaining a preset processing architecture of data, wherein the preset processing architecture at least comprises a computing layer and a storage layer, the computing layer comprises the at least one preset computing module, and the storage layer comprises the at least one preset storage module; and adding the identification information of the at least one preset computing module and the at least one preset storage module through the preset data warehouse.

[0124] Optionally, the processor implements the following steps when executing the program: before predicting the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model, obtaining historical stream data executed by the at least one preset computing module and the at least one preset storage module in a preset time period; identifying an abnormal feature of the historical stream data; and training the preset adaptation model based on the abnormal feature of the historical stream data.

[0125] Optionally, the processor implements the following steps when executing the program: identifying an abnormal keyword in the historical stream data; extracting a time feature of the abnormal keyword in a time dimension; extracting a space feature of the abnormal keyword in a space dimension; and determining the preset adaptation model based on the time feature and the space feature of the abnormal keyword.

[0126] Optionally, the processor implements the following steps when executing the program: before performing data caching of the preset adaptation table by calling stream data and a preset front end through the preset computing module, converting execution code SQL / TableAPI in the preset computing module into a logical execution plan; converting the logical execution plan into a physical execution plan through an optimizer; and generating an executable calling function JobGraph through a code generation technology, wherein the calling function JobGraph is used to configure a calling plan of the preset computing module.

[0127] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0128] The application further provides a computer program product, which, when executed on a data processing device, is adapted to execute a program initialized with the following method steps: obtaining identification information of at least one pre-designed computing module and at least one preset storage module; filling the identification information into a preset adaptation table of a preset data warehouse, wherein the preset data warehouse is used to provide a running environment for adaptation of the pre-designed computing module and the preset storage module, and the preset adaptation table is used to record an adaptation relationship of the pre-designed computing module and the preset storage module; predicting an adaptation probability of the pre-designed computing module and the preset storage module according to a preset adaptation model, and filling the adaptation probability into the preset adaptation table, wherein the preset adaptation model is obtained by training a deep learning algorithm based on a self-attention mechanism; obtaining a data stream processing table by calling stream data and a preset front end for data caching of the preset adaptation table through the pre-designed computing module, wherein the preset front end and the pre-designed computing module establish a mapping relationship in the form of a key-value pair; and analyzing the data stream processing table to realize data anomaly detection.

[0129] Optionally, when executed on a data processing device, the program is adapted to execute a program initialized with the following method steps: before obtaining identification information of at least one pre-designed computing module and at least one preset storage module, obtaining a preset processing architecture of data, wherein the preset processing architecture at least includes: a computing layer and a storage layer, the computing layer includes at least one pre-designed computing module, and the storage layer includes at least one preset storage module; and adding identification information of at least one pre-designed computing module and at least one preset storage module through a preset data warehouse.

[0130] Optionally, when executed on a data processing device, the program is adapted to execute a program initialized with the following method steps: before predicting an adaptation probability of the pre-designed computing module and the preset storage module according to a preset adaptation model, obtaining historical stream data executed by at least one pre-designed computing module and at least one preset storage module in a preset time period; identifying an abnormal feature of the historical stream data; and training the preset adaptation model based on the abnormal feature of the historical stream data.

[0131] Optionally, when executed on a data processing device, the program is adapted to execute a program initialized with the following method steps: identifying an abnormal keyword in historical stream data; extracting a time feature of the abnormal keyword in a time dimension; extracting a space feature of the abnormal keyword in a space dimension; and determining the preset adaptation model based on the time feature and the space feature of the abnormal keyword.

[0132] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: before data caching of the flow data and preset front end of the preset adaptation table is called by a pre-design calculation module, converting execution code SQL / Table API in the pre-design calculation module into a logical execution plan; converting the logical execution plan into a physical execution plan by an optimizer; and generating an executable calling function JobGraph by a code generation technique, wherein the calling function JobGraph is used to configure a calling plan of the pre-design calculation module.

[0133] Those skilled in the art will appreciate that embodiments of the application can be supplied as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer usable program code embodied therein.

[0134] The present application is described below with reference to flow diagrams and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and / or block in the flow diagrams and / or block diagrams, and combinations of flows and / or blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow diagram

[0135] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions means which implement the function specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow diagram

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1steps of a function specified in one or more blocks.

[0137] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0138] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0139] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0141] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon.

[0142] The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A method for detecting data anomalies, characterized in that, include: Obtain the identification information of at least one preset computing module and at least one preset storage module; The identification information is filled into a preset adaptation table of a preset data warehouse. The preset data warehouse adopts data lake technology to provide a running environment for the adaptation of the preset computing module and the preset storage module. The preset adaptation table is used to record the adaptation relationship between the preset computing module and the preset storage module. The adaptation probability of the preset computing module and the preset storage module is predicted according to the preset adaptation model, and the adaptation probability is filled into the preset adaptation table. The preset adaptation model is trained based on a deep learning algorithm with a self-attention mechanism. The preset calculation module calls the streaming data and the preset front-end to perform data caching of the preset adaptation table to obtain the data stream processing table. The preset front-end and the preset calculation module establish a mapping relationship in the form of key-value pairs. The data stream processing table is analyzed to detect data anomalies.

2. The method according to claim 1, characterized in that, Before obtaining the identification information of at least one preset computing module and at least one preset storage module, the method further includes: A preset processing architecture for acquiring data, wherein the preset processing architecture includes at least a computing layer and a storage layer, the computing layer includes at least one preset computing module, and the storage layer includes at least one preset storage module; The identification information is added to at least one of the preset computing modules and at least one of the preset storage modules through the preset data warehouse.

3. The method according to claim 1, characterized in that, Before predicting the compatibility probability between the preset computing module and the preset storage module based on the preset adaptation model, the method further includes: Acquire historical stream data executed by at least one of the preset computing modules and at least one of the preset storage modules within a preset time period; Identify anomalous features in the historical stream data; The preset adaptation model is trained based on the abnormal features of the historical stream data.

4. The method according to claim 3, characterized in that, Training the preset adaptation model based on the abnormal features of the historical stream data includes: Identify abnormal keywords in the historical stream data; Extract the temporal features of the abnormal keywords in the time dimension; Extract the spatial features of the abnormal keywords in the spatial dimension; Based on the temporal and spatial characteristics of the abnormal keywords, the preset adaptation model is determined.

5. The method according to claim 1, characterized in that, Before calling the streaming data and performing data caching of the preset adapted table through the preset calculation module and the preset front end, the method further includes: The execution code in the preset calculation module is converted into a logical execution plan; The optimizer transforms the logical execution plan into a physical execution plan. An executable calling function is generated using code generation technology, wherein the calling function is used to configure the calling plan of the preset computing module.

6. A data anomaly detection device, characterized in that, include: The first acquisition unit is used to acquire the identification information of at least one preset computing module and at least one preset storage module; A storage unit is used to fill the identification information into a preset adaptation table of a preset data warehouse. The preset data warehouse adopts data lake technology and is used to provide an operating environment for the adaptation of the preset computing module and the preset storage module. The preset adaptation table is used to record the adaptation relationship between the preset computing module and the preset storage module. The prediction unit is used to predict the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model, and fill the adaptation probability into the preset adaptation table, wherein the preset adaptation model is trained based on a deep learning algorithm with a self-attention mechanism. The calling unit is used to call the streaming data and the data cache of the preset adaptation table through the preset calculation module to obtain the data stream processing table, wherein the preset front-end and the preset calculation module establish a mapping relationship in the form of key-value pairs; The analysis unit is used to analyze the data stream processing table to detect data anomalies.

7. The apparatus according to claim 6, characterized in that, The device further includes: The second acquisition unit is used to acquire a preset processing architecture of data before acquiring the identification information of at least one preset computing module and at least one preset storage module, wherein the preset processing architecture includes at least a computing layer and a storage layer, the computing layer includes at least one preset computing module, and the storage layer includes at least one preset storage module. An adding unit is used to add the identification information to at least one of the preset computing modules and at least one of the preset storage modules through the preset data warehouse.

8. The apparatus according to claim 6, characterized in that, The device further includes: The third acquisition unit is used to acquire historical stream data of at least one preset computing module and at least one preset storage module executed within a preset time period before predicting the adaptation probability of the preset computing module and the preset storage module according to the preset adaptation model. The first identification unit is used to identify abnormal features of the historical stream data; The training unit is used to train the preset adaptation model based on the abnormal features of the historical stream data.

9. A processor, characterized in that, The processor is used to run a program, wherein the program executes the data anomaly detection method according to any one of claims 1 to 5 when it runs.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data anomaly detection method according to any one of claims 1 to 5.

Images