Feature data anomaly detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a feature data anomaly detection method and device, electronic equipment and storage medium, and belongs to the technical field of data analysis. The method comprises the following steps: analyzing first model feature generation code to obtain feature data; wherein the feature data comprises a source table, a new table, and an association relationship between the source table and the new table; a first directed graph is established according to the feature data; the first directed graph comprises a plurality of nodes, and each node corresponds to one of the source table or the new table; a target node in the first directed graph is detected according to a directed graph algorithm; target feature data corresponding to the target node is extracted, and the target feature data is sent to a target object; and an anomaly detection result of the target feature data returned by the target object is received. The device comprises an analysis module, a directed graph module, a first detection module and a first detection module. The embodiment of the application can perform table-level detection on feature data, which is beneficial to improving detection accuracy and detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and particularly relates to a feature data anomaly detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] At present, model feature data based on an AI model is increasing day by day, and data management based on big data has a problem of unclear data relationship, so model feature data is managed through data task scheduling. However, the granularity of task scheduling is often too coarse, and the task scheduling relationship depends on manual configuration, and there are often omissions, resulting in low accuracy and low efficiency of feature data anomaly detection. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a feature data anomaly detection method and device, electronic equipment and a storage medium, which can detect feature data at a table level, and is beneficial to improve detection accuracy and detection efficiency.

[0004] To achieve the above purpose, a first aspect of the embodiments of the present application provides a feature data anomaly detection method, which comprises the following steps:

[0005] Analyzing a first model feature generation code to obtain feature data; wherein the feature data comprises a source table, a new table, and an association relationship between the source table and the new table;

[0006] According to the feature data, a first directed graph is established; the first directed graph comprises a plurality of nodes, and each node corresponds to one of the source table or the new table;

[0007] According to a directed graph algorithm, a target node in the first directed graph is detected, wherein the target node comprises at least one of an isolated node, a ring node or a terminal node;

[0008] Extracting target feature data corresponding to the target node, and sending the target feature data to a target object, and receiving an anomaly detection result of the target feature data returned by the target object.

[0009] In the embodiments of the present application, the feature data is obtained by analyzing the first model feature generation code, wherein the feature data includes a source table, a new table, and an association relationship between the source table and the new table; a first directed graph is established according to the feature data, the first directed graph includes a plurality of nodes, each node corresponds to one of the source table or the new table; the model feature data is associated with the directed graph, so as to facilitate detection and management of the model feature data by the algorithm of the directed graph; a target node in the first directed graph is detected according to the directed graph algorithm, wherein the target node includes at least one of an isolated node, a ring node, or a terminal node; target feature data corresponding to the target node is extracted and sent to a target object, and an abnormal detection result of the target feature data returned by the target object is received; the feature data corresponding to the target node (at least one of the isolated node, the ring node, or the terminal node) in the directed graph is usually abnormal, and needs to be observed and detected, therefore, the abnormal node is detected to detect the feature data. Through the above method, the feature data can be detected at the table level, which is beneficial to improve the detection accuracy and detection efficiency.

[0010] In some embodiments, the first model feature generation code includes an SQL statement, and the analyzing the first model feature generation code to obtain the feature data includes:

[0011] A first keyword group and a second keyword group are obtained according to a preselected setting; the first keyword group includes a plurality of first keywords, and the second keyword group includes a plurality of second keywords;

[0012] The source table is obtained by matching the SQL statement with the first keyword;

[0013] The new table is obtained by matching the SQL statement with the second keyword.

[0014] In some embodiments, the first directed graph is established according to the feature data, including:

[0015] A first node is established according to the source table;

[0016] A second node is established according to the new table;

[0017] An edge between the first node and the second node is established according to the association relationship; wherein the direction of the edge is from the first node to the second node;

[0018] A first directed graph is established according to the first node, the second node, and the edge.

[0019] In some embodiments, the method further comprises the following steps:

[0020] According to a subsequent node algorithm of the first directed graph, a path of a target node is determined, and a dependency relationship is established;

[0021] The dependency relationship is sent to a target object, and an anomaly detection result of the target feature data returned by the target object is received.

[0022] In some embodiments, the method further comprises the following steps:

[0023] The generation time of the end node is obtained;

[0024] If the generation time is greater than a preset time threshold, the end node is determined as a target node.

[0025] In some embodiments, the method further comprises the following steps:

[0026] Whether the first directed graph has a loop is determined by a directed graph depth traversal algorithm;

[0027] If the first directed graph has a loop, the nodes in the loop are determined as target nodes.

[0028] In some embodiments, the method further comprises the following steps:

[0029] The first model feature generation code is parsed in time, and a first directed graph corresponding to the first model feature generation code is established;

[0030] The part of the first directed graph that coincides with a second directed graph of a second model feature generation code is filtered, and the first directed graph is obtained;

[0031] The generation time of the second model feature generation code is earlier than the generation time of the first model feature generation code, and the difference between the generation time of the second model feature generation code and the generation time of the first model feature generation code is less than a second preset time threshold.

[0032] To achieve the above object, a second aspect of the embodiment of the present application provides an anomaly detection device of feature data, which comprises:

[0033] A parsing module is configured to parse a first model feature generation code to obtain feature data; wherein the feature data comprises a source table, a new table, and an association relationship between the source table and the new table;

[0034] A directed graph module is configured to establish a first directed graph according to the feature data; the first directed graph comprises a plurality of nodes, and each node corresponds to one of the source table or the new table;

[0035] The first detection module is configured to detect a target node in the first directed graph according to a directed graph algorithm, wherein the target node includes at least one of an isolated node, a ring node, or a terminal node.

[0036] The second detection module is configured to extract target feature data corresponding to the target node, and send the target feature data to a target object, and receive an abnormality detection result of the target feature data returned by the target object.

[0037] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the method of the first aspect.

[0038] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium for computer readable storage, wherein the storage medium stores one or more computer programs, and the one or more computer programs are executable by one or more processors to realize the method of the first aspect.

[0039] The feature data abnormality detection method and device, electronic device and storage medium provided by the present application can obtain feature data by analyzing first model feature generation code, wherein the feature data includes a source table, a new table and an association relationship between the source table and the new table; a first directed graph is established according to the feature data, the first directed graph includes a plurality of nodes, each node corresponds to one of the source table or the new table; the model feature data is associated with the directed graph, so as to facilitate detection and management of the model feature data by using the algorithm of the directed graph; a target node in the first directed graph is detected according to a directed graph algorithm, wherein the target node includes at least one of an isolated node, a ring node or a terminal node; target feature data corresponding to the target node is extracted, and the target feature data is sent to a target object, and an abnormality detection result of the target feature data returned by the target object is received; the feature data corresponding to the target node (at least one of the isolated node, the ring node or the terminal node) in the directed graph is usually abnormal, and needs to be observed and detected; therefore, the abnormal node is detected to detect the feature data. Through the above method, the feature data can be detected at the table level, which is beneficial to improving the detection accuracy and the detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1is a flowchart of an embodiment of step S101 in the feature data anomaly detection method provided by the present application;

[0041] Figure 2 is Figure 1 is a flowchart of an embodiment of step S101 in the feature data anomaly detection method provided by the present application;

[0042] Figure 3 is Figure 1 is a flowchart of another embodiment of step S102 in the feature data anomaly detection method provided by the present application;

[0043] Figure 4 is a flowchart of another embodiment of the feature data anomaly detection method provided by the present application;

[0044] Figure 5 is a structural schematic diagram of the feature data anomaly detection apparatus provided by the present application;

[0045] Figure 6 is a hardware structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0047] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0049] First, the several terms involved in the present application are analyzed:

[0050] Artificial intelligence (AI): is a new technical science of researching and developing theory, method, technology and application system for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence, the research in this field includes robots, language recognition, image recognition, natural language processing and expert system, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0051] Graph: refers to a data form composed of many nodes (also known as vertices) connected with each other, where the nodes can be entities such as people and institutions, and the connection (called edge) between the nodes represents a certain relationship (such as friendship, affiliation, etc.); a graph can have only one node and one edge (called a single graph), or multiple nodes or multiple edges (called a heterogeneous graph), where the edges in the graph can be directed edges (called a directed graph) or undirected edges (called an undirected graph).

[0052] Big data: refers to a collection of data that cannot be captured, managed and processed within a certain time range by conventional software tools, and is a massive, high-growth and diversified information asset that requires new processing mode to have stronger decision-making, insight discovery and process optimization capabilities. With the advent of the cloud era, big data has attracted more and more attention, and big data requires special technology to effectively process large amounts of data over time. The technologies suitable for big data include large-scale parallel processing database, data mining, distributed file system, distributed database, cloud computing platform, Internet and scalable storage system.

[0053] Hadoop: is a software framework capable of distributed processing of large amounts of data. Hadoop includes four modules of Common, HDFS (Hadoop Distributed File System), YARN (Yet Another Resource Negotiator) and MapReduce, wherein Common: is a common tool that can support other modules; HDFS is a distributed file system for providing high-throughput access performance; YARN is a framework for job scheduling and cluster resource management; MapReduce is a data parallel computing framework, abbreviated as MR.

[0054] Hive: is a data warehouse tool based on Hadoop, which can map structured data files to a database table and provide SQL query function, which can convert SQL statement into MapReduce task for running. Hive management of data warehouse includes two aspects: one is the management of metadata, two is the management of data.

[0055] Spark: Apache Spark is a fast general-purpose computing engine designed for large-scale data processing. Spark is an open source Hadoop MapReduce-like general-purpose parallel framework.

[0056] SQL: is a programming language with special purposes, which is a database query and programming language, used to access data and query, update and manage relational database system.

[0057] With the expansion of AI service range in business processes, the number and complexity of models are increasing, and there will be difficulties in feature management of models. At present, the model features based on business demands come from business systems or user clients, and the original feature data comes from HADOOP and HIVE storage of distributed big data systems. However, there is a lack of data management based on big data in the industry, which leads to unclear data usage relationship, omissions when upstream modification is coordinated with downstream, inability to find redundant data, difficulty in repairing abnormal feature data, and other problems.

[0058] Based on the above problems, data task scheduling is usually used for management at present, but the granularity of task scheduling is too coarse, a task may contain processing and production of multiple feature tables, and the task scheduling relationship depends on manual configuration, often with omissions. Feature data anomalies will cause inaccurate model training and online prediction, thereby affecting business use. Therefore, the current generation of model features needs to start from the relationship at the table level, and needs to be managed uniformly and completely to ensure the safe and stable operation of feature data and models.

[0059] Based on this, the embodiment of the application provides an exception detection method and device for feature data, an electronic device and a storage medium. The feature data is obtained by analyzing a first model feature generation code, wherein the feature data includes a source table, a new table and an association relationship between the source table and the new table; a first directed graph is established according to the feature data, the first directed graph includes a plurality of nodes, and each node corresponds to one of the source table or the new table; the model feature data is associated with the directed graph, so that the model feature data is detected and managed by the algorithm of the directed graph; a target node in the first directed graph is detected according to the directed graph algorithm, wherein the target node includes at least one of an isolated node, a ring node or a terminal node; target feature data corresponding to the target node is extracted, and the target feature data is sent to a target object; and an exception detection result of the target feature data returned by the target object is received. The feature data corresponding to the target node (at least one of the isolated node, the ring node or the terminal node) in the directed graph is usually abnormal, and needs to be observed and detected. Therefore, the feature data is detected to detect the exception of the feature data. Through the above method, the feature data can be detected at the table level, which is beneficial to improve the detection accuracy and the detection efficiency.

[0060] The exception detection method and device, the electronic device and the storage medium provided by the embodiment of the application are specifically described through the following embodiments. First, the exception detection method in the embodiment of the application is described.

[0061] The embodiment of the application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain the best results.

[0062] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and the like. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning and the like.

[0063] The feature data anomaly detection method provided in the embodiments of the present application relates to the technical field of data analysis. The feature data anomaly detection method provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as a separate physical server, can also be configured as a server cluster or a distributed system formed by multiple physical servers, and can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms; and the software can be an application that implements the feature data anomaly detection method, but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0065] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information, etc. related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, etc. After obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally run is obtained.

[0066] Figure 1 is an optional flowchart of the feature data anomaly detection method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to steps S101 to S104.

[0067] In step S101, the first model feature generation code is parsed to obtain feature data; wherein the feature data includes a source table, a new table, and an association relationship between the source table and the new table.

[0068] In some embodiments, the feature raw data is derived from HADOOP and HIVE of a distributed big data system, and thus the first model feature generation code can be a HIVE SQL code, a log code, a SPARK SQL code, or the like, and the application does not limit the specific form of the first model feature generation code. By parsing the first model feature generation code, the feature data in the first model feature generation code is obtained for subsequent directed graph construction and feature data detection. In some embodiments, the feature data includes a source table, a new table, and an association relationship between the source table and the new table. In data management, the most important thing is to extract the source table (source) and the new table (target), and through data processing and changes of the source table, the feature data of the new table is obtained, and thus the association relationship between the source table and the new table is usually from the source table to the new table through processing. Through the source table, the new table, and the association relationship between the source table and the new table, the subsequent directed graph establishment process is facilitated.

[0069] In step S102, a first directed graph is established according to the feature data; the first directed graph includes a plurality of nodes, and each node corresponds to one of the source table or the new table.

[0070] In some embodiments, the directed graph is established to represent the association relationship between the feature data. Exemplarily, the node in the first directed graph corresponds to one of the source table or the new table in the feature data; meanwhile, the first directed graph also includes an association relationship between the nodes, which corresponds to the association relationship between the source table and the new table in the feature data. Through the nodes and the association relationship between the nodes, the feature data is represented through the first directed graph, so as to perform anomaly detection or data management on the feature data through a related algorithm of the directed graph.

[0071] In some embodiments, the source of the new table in the feature data can be multiple, and thus the last node of the node in the first directed graph can be one or multiple, i.e., the source of the node can be two nodes in the last group of nodes. Therefore, through the first directed graph, the data association relationship in the feature data can be visually represented, which is beneficial to improving the efficiency of data management.

[0072] In step S103, a target node in the first directed graph is detected according to a directed graph algorithm; wherein the target node includes at least one of an isolated node, a ring node, or a terminal node.

[0073] In some embodiments, the feature data is managed and detected by a directed graph algorithm, and a target node in the first directed graph is detected and identified. In some possible embodiments, the target node can be an isolated node, the target node can also be a ring node, and the target node can also be a terminal node. For the isolated node, the ring node or the terminal node in the directed graph, it can be considered as a node that needs to be highly concerned, i.e., the target node. Exemplarily, the isolated node has no source and no destination; therefore, the feature data corresponding to the isolated node can also have a data isolation problem, and the feature data is not referenced or processed, which indicates that the feature data can be forgotten data or unsuccessfully deleted data; the node is marked as the target node, i.e., the feature data corresponding to the node is marked, so as to facilitate the subsequent abnormality detection process.

[0074] In step S104, the target feature data corresponding to the target node is extracted, and the target feature data is sent to the target object, and an abnormality detection result of the target feature data transmitted back by the target object is received.

[0075] In some embodiments, the target feature data corresponding to the target node, i.e., the data that needs to be monitored and processed, is sent to the target object, and an abnormality detection result of the target feature data transmitted back by the target object is received. Exemplarily, the target feature data corresponding to the isolated node is sent to the target object, so that the target object performs abnormality judgment on the target feature data, and gives a judgment conclusion and a processing measure. For example, for the target feature data that is forgotten data, the associated relationship corresponding to the data is re-established to perfect the associated relationship of the target node, and the isolated node is re-established to remove the isolated node. For the data that is unsuccessfully deleted data, the data is deleted to release the memory, i.e., the target node corresponding to the target feature data is also deleted, and the isolated node will be deleted. Of course, the above is an exemplary example, and a person skilled in the art can detect and process the target node and the corresponding target feature data according to actual needs, so as to improve the accuracy of the feature data, and further improve the accuracy of the artificial intelligence model.

[0076] The steps S101 to S104 shown in the embodiments of the present application are used to obtain feature data by analyzing the first model feature generation code, wherein the feature data includes the source table, the new table, and the association relationship between the source table and the new table; a first directed graph is established according to the feature data, the first directed graph includes a plurality of nodes, and each node corresponds to one of the source table or the new table; the model feature data is associated with the directed graph, so as to facilitate detection and management of the model feature data through the algorithm of the directed graph; a target node in the first directed graph is detected according to the directed graph algorithm, wherein the target node includes at least one of an isolated node, a ring node, or a terminal node; target feature data corresponding to the target node is extracted, and the target feature data is sent to a target object; and an abnormal detection result of the target feature data returned by the target object is received. The feature data corresponding to the target node (at least one of the isolated node, the ring node, or the terminal node) in the directed graph is usually abnormal, and needs to be observed and detected. Therefore, the abnormal node is detected to detect the abnormality of the feature data. Through the above method, the feature data can be detected at the table level, which is beneficial to improve the detection accuracy and the detection efficiency.

[0077] In some embodiments, as shown in Figure 2 The first model feature generation code includes a SQL statement, and the step S101 can include but is not limited to steps S201 to S203:

[0078] In step S201, a first keyword group and a second keyword group are obtained, the first keyword group includes a plurality of first keywords, and the second keyword group includes a plurality of second keywords.

[0079] In step S202, the source table is obtained by matching the SQL statement with the first keyword.

[0080] In step S203, the new table is obtained by matching the SQL statement with the second keyword.

[0081] In some embodiments, the processing procedure of the source table to the new table may involve multi-layer transformation due to processing efficiency and data management, etc., and thus there may be a final feature data generated by multiple SQLs and multiple tasks; that is, the new table may be derived from multiple source tables. For a single SQL, code parsing is performed through keyword capturing to obtain the source table and the new table. Exemplarily, code parsing of the source table is performed through a first keyword group, and code parsing of the new table is performed through a second keyword group. The first keyword group includes a plurality of first keywords, and the second keyword group includes a plurality of second keywords. Exemplarily, the first keyword can be any one of FROM, JOIN, and WITH; and the second keyword can be any one of CREATE and INSERT. Of course, the above first keyword and second keyword are exemplary examples, and a person skilled in the art can set appropriate first keywords and second keywords according to requirements to match the SQL statement or other forms of first model feature generation code to obtain the source table and the new table. Through the source table and the new table, a directed graph between the feature data is established to detect the anomaly of the feature data.

[0082] Referring to Figure 3 In some embodiments, step S102 can include but is not limited to steps S301 to S304:

[0083] Step S301, establishing a first node according to the source table;

[0084] Step S302, establishing a second node according to the new table;

[0085] Step S303, establishing an edge between the first node and the second node according to the association relationship; wherein the direction of the edge is from the first node to the second node;

[0086] Step S304, establishing a first directed graph according to the first node, the second node, and the edge.

[0087] In some embodiments, a directed graph is established according to the source table, the new table, and the relationship between the two in the feature data. A first node corresponding to the source table is established according to the source table; a second node corresponding to the new table is established according to the new table; an edge between the first node and the second node is established according to the association relationship; wherein, since the data flow is from the source table to the new table, the direction of the edge is from the first node to the second node; a first directed graph is established according to the first node, the second node, and the edge. After all the source tables and new tables in the feature data are represented by the first node, the second node, and the edge in the directed graph, a directed graph corresponding to the feature data is obtained.

[0088] Exemplarily, after the source table and the new table are parsed, the data processing flow is from the source table to the new table (source-target), and therefore the directed graph includes: a node Node (source1, source2, …, sourceN, target) and a set of edges Edge (<s1, t>, <s2, t>, …, <sn, t>), and the direction of the edges is source->target. After the code parsing of the first model feature is performed, a global node Node_set and a corresponding global edge Edge_set of the feature data are obtained, and a whole directed graph (Direct Graph) related to the feature data is constructed.

[0089] In some embodiments, the method further includes steps S401 to S402:

[0090] Step S401, determining a path of a target node according to a successors algorithm of the first directed graph, and establishing a dependency relationship;

[0091] Step S402, sending the dependency relationship to a target object, and receiving an abnormality detection result of the target feature data returned by the target object.

[0092] In some embodiments, a successors algorithm of a directed graph algorithm is used to find all paths of a target data table (target), and a table-level dependency relationship is constructed. The above dependency relationship is sent to a target object, and an abnormality detection result of the target feature data returned by the target object is received. For the problem that the dependency relationship of the feature data is unknown, the above method can sort out the dependency relationship and send the result to the target object, so as to manage and detect the dependency relationship, so as to find out whether the feature data is abnormal. Exemplarily, the acquisition of the dependency relationship can also be used to analyze whether the corresponding subtask is modified and offline after the single table is offline.

[0093] In some embodiments, the above method further includes steps S501 to S502:

[0094] Step S501, acquiring a generation time of an end node;

[0095] Step S502, if the generation time is greater than a preset time threshold, determining that the end node is a target node.

[0096] In some embodiments, the end node of the directed graph, i.e., the end node without subsequent data referring to the node, is the node finally generated by the main model feature, and the generation time of the node is also a quantity to be detected. Exemplarily, if the generation time is too long, it indicates that there is an abnormality in the generation process or the intermediate node of the node, and therefore, the end node with a generation time greater than a preset time threshold is determined as a target node, which is sent to a target object for further detection and processing to find and determine the abnormality of the feature data.

[0097] In some embodiments, the target node can also be obtained by detecting the upstream task node. Exemplarily, the upstream node (i.e., the predecessors process) is traced back from the new table, and the state of the upstream table data is traversed to find an abnormal node. Specifically, the longest generation path of the end node is obtained, which is compared with a preset path to determine whether the end node belongs to the target node. Those skilled in the art can select the specific logic of target node detection and the specific value of the preset threshold according to the needs to detect and judge the abnormality of the feature data.

[0098] In some embodiments, the above method further includes steps S601 to S602:

[0099] Step S601, determining whether there is a loop in the first directed graph by using a directed graph depth traversal algorithm;

[0100] Step S602, if there is a loop in the first directed graph, determining the nodes in the loop as target nodes.

[0101] In some embodiments, for data tasks, a directed acyclic scheduling path should be used to avoid loops, and abnormal tasks in the loop can cause link to be stuck or fall into an infinite loop. Therefore, the depth traversal of the directed graph is used to determine whether all subsequent child nodes of each depth node have appeared before in the previous depth to determine whether there is a loop. If there is a loop, the nodes in the loop are determined as target nodes for the next step of abnormality detection to discover and handle the abnormality in time.

[0102] In some embodiments, the above method further includes steps S701 to S703:

[0103] Step S701, parsing the first model feature generation code at a fixed time to establish a first directed graph corresponding to the first model feature generation code;

[0104] Step S702, filtering the part of the first directed graph that coincides with a second directed graph of a second model feature generation code to obtain the first directed graph;

[0105] The generation time of the second model feature generation code is earlier than the generation time of the first model feature generation code, and the difference between the generation time of the second model feature generation code and the generation time of the first model feature generation code is less than a second preset time threshold.

[0106] In some embodiments, the above-mentioned anomaly detection method can be periodic anomaly detection on feature data. Therefore, a timing strategy can be set as needed. For example, anomaly detection is performed on feature data every two hours; or different detection interval intervals are set according to different time periods, and anomaly detection is performed on feature data; or different detection interval intervals are set according to the use frequency of the artificial intelligence model, and anomaly detection is performed on feature data.

[0107] In some embodiments, the generation time of the second model feature generation code is earlier than the generation time of the first model feature generation code, and the difference between the generation time of the second model feature generation code and the generation time of the first model feature generation code is less than a second preset time threshold. Illustratively, the feature data of the latest 2 hours (for example, 11:00-13:00) is the first directed graph, and the historical record stored locally is the second directed graph, which can be the feature data earlier than the latest 2 hours (for example, 9:00-11:00). Before performing anomaly access detection, the part of the first feature data corresponding to the first directed graph that coincides with the third directed graph corresponding to the third feature data can be filtered, the access behavior that occurred in the latest 2 hours during the period earlier than the latest 2 hours is screened, and the filtered directed graph is obtained. Through filtering, the detection load is reduced, the detection efficiency is improved, and the detection time is shortened.

[0108] See Figure 4The embodiment of the application provides a feature data anomaly detection method and device, an electronic device and a storage medium, a feature data is obtained by analyzing a first model feature generation code, wherein the feature data comprises a source table, a new table and an association relationship between the source table and the new table; a first directed graph is established according to the feature data, the first directed graph comprises a plurality of nodes, each node corresponds to one of the source table or the new table; the model feature data is associated with the directed graph, so that the model feature data is detected and managed through the algorithm of the directed graph; a target node in the first directed graph is detected according to the directed graph algorithm, wherein the target node comprises at least one of an isolated node, a ring node or a terminal node; target feature data corresponding to the target node is extracted, and the target feature data is sent to a target object; an anomaly detection result of the target feature data returned by the target object is received; the feature data corresponding to the target node (at least one of the isolated node, the ring node or the terminal node) in the directed graph is usually abnormal, and needs to be observed and detected, so that the feature data is detected to perform anomaly detection. Through the above method, the feature data can be detected at the table level, which is beneficial to improving the detection accuracy and the detection efficiency.

[0109] Meanwhile, after the method provided by the application is used, the model feature generation is controlled in the process, and the integrity and efficiency of data management are improved. The data online and offline, abnormal judgment can be completely covered. Meanwhile, in the cost control optimization, the method realizes peak-shaving execution of high resource consumption task link by analyzing the key path of the task, and the resource dispersion of the overall model feature data task on the bottleneck task, so that less resources complete the original task. In the function offline and function attribution, the overall function attribution is divided through the end data backtracking, which plays a data module management function.

[0110] Meanwhile, the method realizes a large amount of saving of manpower and improvement of efficiency, and the number of manpower investment is averagely reduced by 50% and the time efficiency is averagely improved by 30% in abnormal positioning optimization and function offline analysis, which has a broad application prospect.

[0111] Please refer to Figure 5 The embodiment of the application further provides a feature data anomaly detection device, which can realize the feature data anomaly detection method.

[0112] The analysis module 501 is configured to analyze a first model feature generation code to obtain feature data; wherein the feature data comprises a source table, a new table and an association relationship between the source table and the new table;

[0113] The directed graph module 502 is configured to establish a first directed graph according to the feature data; the first directed graph comprises a plurality of nodes, each node corresponds to one of the source table or the new table;

[0114] The first detection module 503 is configured to detect a target node in the first directed graph according to a directed graph algorithm, wherein the target node includes at least one of an isolated node, a loop node or a terminal node.

[0115] The second detection module 504 is configured to extract target feature data corresponding to the target node, and send the target feature data to a target object, and receive an abnormality detection result of the target feature data returned by the target object.

[0116] The specific implementation of the feature data abnormality detection apparatus is basically the same as that of the above-mentioned specific embodiment of the feature data abnormality detection method, and will not be repeated here.

[0117] The embodiments of the present application further provide an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the feature data abnormality detection method. The electronic device can be any smart terminal including a tablet computer, a vehicle-mounted computer, etc.

[0118] Please refer to Figure 6 , Figure 6 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0119] The processor 601 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0120] The memory 602 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device or a RAM (Random Access Memory), etc. The memory 602 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 602 and are called and executed by the processor 601 to realize the feature data abnormality detection method of the embodiments of the present application.

[0121] The input / output interface 603 is used to realize information input and output.

[0122] The communication interface 604 is configured to realize the communication interaction between the device and other devices. The communication can be realized in a wired manner (for example, a USB, a network cable, and the like) or in a wireless manner (for example, a mobile network, WIFI, Bluetooth, and the like).

[0123] The bus 605 is configured to transmit information between various components (for example, the processor 601, the memory 602, the input / output interface 603, and the communication interface 604) of the device.

[0124] The processor 601, the memory 602, the input / output interface 603, and the communication interface 604 are connected to each other through the bus 605.

[0125] The embodiment of the present application further provides a computer readable storage medium, which is configured to store a computer readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to realize the above-mentioned feature data anomaly detection method.

[0126] The memory is a non-transitory computer readable storage medium, and can be configured to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0127] The feature data anomaly detection method and device, electronic equipment and storage medium provided by the present application, by analyzing the first model feature generation code, obtain feature data, wherein the feature data includes a source table, a new table and an association relationship between the source table and the new table; according to the feature data, a first directed graph is established, the first directed graph includes a plurality of nodes, each node corresponds to one of the source table or the new table; by establishing an association relationship between the model feature data and the directed graph, it is convenient to detect and manage the model feature data through the algorithm of the directed graph; according to the directed graph algorithm, a target node in the first directed graph is detected, wherein the target node includes at least one of an isolated node, a ring node or a terminal node; the target feature data corresponding to the target node is extracted, and the target feature data is sent to a target object, and the anomaly detection result of the target feature data returned by the target object is received; the feature data corresponding to the target node (at least one of the isolated node, the ring node or the terminal node) in the directed graph is usually abnormal, and needs to be observed and detected. Therefore, the feature data is detected to detect the anomaly of the feature data. Through the above method, the feature data can be detected at the table level, which is beneficial to improve the detection accuracy and detection efficiency.

[0128] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0129] Those skilled in the art can understand that, Figures 1-6 The technical solutions shown in the foregoing embodiments do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the drawings, or combine certain steps or different steps.

[0130] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0131] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0132] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect between similar objects.

[0133] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0134] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0135] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0136] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0137] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0138] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of anomaly detection of feature data, characterized by, The method comprises: parsing the first model feature generation code to obtain feature data; wherein the feature data comprises a source table, a new table, and an association relationship between the source table and the new table; establishing a first directed graph according to the feature data; the first directed graph comprises a plurality of nodes, each of which corresponds to one of the source table or the new table; detecting a target node in the first directed graph according to a directed graph algorithm; wherein the target node comprises at least one of an isolated node, a ring node, or a terminal node; extracting target feature data corresponding to the target node and sending the target feature data to a target object, and receiving an abnormality detection result of the target feature data returned by the target object; wherein, according to the feature data, the first directed graph is established, comprising: establishing a first node according to the source table; establishing a second node according to the new table; establishing an edge between the first node and the second node according to the association relationship; wherein the direction of the edge is from the first node to the second node; establishing a first directed graph according to the first node, the second node, and the edge; The method further comprises: determining whether the first directed graph has a ring through a directed graph depth traversal algorithm; if the first directed graph has a ring, determining the nodes in the ring as target nodes.

2. The abnormality detection method according to claim 1, characterized by, The first model feature generation code comprises an SQL statement, and the parsing of the first model feature generation code to obtain the feature data comprises: obtaining a preselected first keyword group and a second keyword group; the first keyword group comprises a plurality of first keywords, and the second keyword group comprises a plurality of second keywords; matching the SQL statement with the first keywords to obtain the source table; matching the SQL statement with the second keywords to obtain the new table.

3. The abnormality detection method according to claim 1, characterized by, The method further comprises the following steps: determining the path of the target node and establishing a dependency relationship according to a subsequent node algorithm of the first directed graph; sending the dependency relationship to a target object and receiving an abnormality detection result of the target feature data returned by the target object.

4. The abnormality detection method according to claim 1, characterized by, The method further comprises the following steps: obtaining the generation time of the terminal node; if the generation time is greater than a pre-set time threshold, determining the terminal node as a target node.

5. The abnormality detection method according to claim 1, characterized by, The method further comprises: parsing the first model feature generation code at regular intervals to establish a first directed graph corresponding to the first model feature generation code; filtering the part of the first directed graph that coincides with a second directed graph of a second model feature generation code to obtain the first directed graph; wherein, the generation time of the second model feature generation code is earlier than the generation time of the first model feature generation code, and the difference between the generation time of the second model feature generation code and the generation time of the first model feature generation code is less than a second pre-set time threshold.

6. An anomaly detection device of feature data, characterized by, The device comprises: a parsing module for parsing the first model feature generation code to obtain feature data; wherein the feature data comprises a source table, a new table, and an association relationship between the source table and the new table; a directed graph module, configured to establish a first directed graph according to the feature data, the first directed graph comprising a plurality of nodes, each of the nodes corresponding to one of the source table or the new table; a first detection module, configured to detect a target node in the first directed graph according to a directed graph algorithm, wherein the target node comprises at least one of an isolated node, a loop node or a terminal node; a second detection module, configured to extract target feature data corresponding to the target node, and send the target feature data to a target object, and receive an abnormality detection result of the target feature data returned by the target object; wherein the directed graph module is configured to establish the first directed graph according to the feature data, comprising: establishing a first node according to the source table; establishing a second node according to the new table; establishing an edge between the first node and the second node according to the association relationship, wherein a direction of the edge is from the first node to the second node; establishing a first directed graph according to the first node, the second node and the edge; the apparatus is further configured to: determine whether the first directed graph has a loop by using a directed graph depth-first search algorithm; if the first directed graph has a loop, determine that a node in the loop is a target node.

7. An electronic device, comprising: The electronic device comprises a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program, when executed by the processor, realizes the steps of the method according to any one of claims 1 to 5.

8. A computer readable storage medium for computer readable storage, characterized in that, The storage medium stores one or more computer programs, and the one or more computer programs are executable by one or more processors to realize the steps of the method according to any one of claims 1 to 5.

Images