Abnormal data detection method and device, equipment, medium and product

Abstract

The invention discloses an abnormal data detection method, device and equipment, a medium and a product, and relates to the technical field of data analysis. The method comprises the following steps: traversing a system database to obtain system data, and for each data table in the system data, associating a data table name of the data table with a data field name in the data table to generate a target data dictionary; performing grammatical analysis on the system data, and constructing a data link set in combination with a path search algorithm; and performing real-time detection on the system database based on the target data dictionary and the data link set. By adopting the technical scheme, the problem of how to quickly position the abnormal data is solved, the database is monitored in real time based on the data dictionary and the data link set, and the abnormal data and the influence range thereof are automatically monitored in real time.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to an anomaly detection method, apparatus, equipment, medium, and product. Background Technology

[0002] With the continuous development of the system and the constant iteration of business scenarios, the amount of data in the database is constantly increasing. Under these circumstances, monitoring database anomalies becomes particularly important.

[0003] Current methods for monitoring abnormal data rely solely on single-point data statistical feature detection, lacking business semantic understanding and resulting in isolated anomaly detection. Meanwhile, existing data lineage analysis tools depend on static metadata, making it difficult to reflect the propagation impact of data changes in a timely manner, leading to a lag in impact assessment. Furthermore, detection rules are highly dependent on manual maintenance, resulting in slow response times and an inability to adapt to high-frequency data change scenarios.

[0004] Therefore, there is an urgent need for a data detection method to automatically monitor abnormal data, quickly locate the scope of impact, and improve the stability of system operation. Summary of the Invention

[0005] This invention provides an abnormal data detection method, apparatus, equipment, medium, and product. By adopting this technical solution, the problem of how to quickly locate abnormal data is solved. Based on the data dictionary and data link set, the database is monitored in real time, realizing automatic real-time monitoring of abnormal data and its impact range.

[0006] According to one aspect of the present invention, an abnormal data detection method is provided, comprising:

[0007] The system database is traversed to obtain system data. For each data table in the system data, the table name and the data field names in the table are associated to generate a target data dictionary.

[0008] The system data is subjected to syntactic analysis, and a data link set is constructed by combining it with a path search algorithm;

[0009] The system database is monitored in real time based on the target data dictionary and the data link set.

[0010] According to another aspect of the present invention, an abnormal data detection device is provided, comprising:

[0011] The target data dictionary generation module is used to traverse the system database to obtain system data, and for each data table in the system data, associate the data table name and the data field name in the data table to generate a target data dictionary;

[0012] The data link set construction module is used to perform syntactic analysis on the system data and construct a data link set in combination with a path search algorithm;

[0013] The detection module is used to perform real-time detection on the system database based on the target data dictionary and the data link set.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the abnormal data detection method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the abnormal data detection method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the abnormal data detection method according to any embodiment of the present invention.

[0020] The technical solution of this invention addresses the problem of how to quickly locate abnormal data. It performs real-time monitoring of the database based on a data dictionary and data link set, thereby achieving automatic real-time monitoring of abnormal data and its impact range.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a flowchart of an abnormal data detection method provided by an embodiment of the present invention;

[0024] Figure 2 This is a flowchart of an abnormal data detection method provided by an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of a method for determining a data link set applicable to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an abnormal data detection device according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the abnormal data detection method of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of system data involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] Figure 1This invention provides a flowchart of an abnormal data detection method. This embodiment is applicable to situations where abnormal monitoring of system data in a system database is required. The method can be executed by an abnormal data detection device, which can be implemented in hardware and / or software and can be configured in a server. Figure 1 As shown, the method includes:

[0032] S110. Traverse the system database to obtain system data. For each data table in the system data, associate the data table name with the data field names in the data table to generate the target data dictionary.

[0033] Among them, the system database is the core of the information system center for data storage, including different business scenarios; system data can be business data; the data table name is the name of the data table, and the data field name in the data table can be the name of the data field; the target data dictionary is a structured document used to systematically describe the definition and attributes of each data in the system data.

[0034] Specifically, the system database is traversed and queried to obtain system data, which may include at least one data table; for each data table in the system data, a mapping relationship is established between the data table name and the data field names in the data table, and a target data dictionary is generated based on the mapping relationship and the attribute information of the fields.

[0035] Optionally, for each data table in the system data, the table name and the data field names within that table are associated to generate a target data dictionary, including:

[0036] The system data is filtered according to preset field thresholds to obtain the target data;

[0037] Use the combination of the table name and field name to which the target data belongs as the key name;

[0038] Use the attribute values ​​corresponding to the data table as the key values;

[0039] Insert specific symbols between the key name and the key value to obtain the initial data dictionary;

[0040] The target data dictionary is obtained in response to the supplementary operations on the initial data dictionary.

[0041] The target data refers to the filtered data tables and the data field content within each table; the attribute value is the data field attribute information, which may include data type and constraints; the specific character is the character that separates the key name and key value; the initial data dictionary is a key-value pair data dictionary; the supplementary operation involves obtaining technical personnel to review and supplement the content of the initial data dictionary; the target data dictionary can be a baseline version of the data dictionary. It should be noted that the field threshold is preset, preferably 10 in this embodiment; relevant technical personnel can adjust it according to actual needs, and this embodiment does not specifically limit it.

[0042] Specifically, it can iterate through each data field in the data table, count the number of unique values ​​in each data field, and filter based on preset field thresholds. Data fields with values ​​less than the field threshold are counted, and their table names and field names are combined to form key names, with the attribute values ​​corresponding to the field names used as key values. Key names or key values ​​are separated by specific symbols to form an initial data dictionary. In response to manual review and supplementation of the initial data dictionary by technical personnel, a baseline version of the data dictionary is obtained by supplementing the initial data dictionary with the supplementary content, which serves as the target data dictionary.

[0043] For example, in a data table named TAB FEECHARGEDETL and a field named CHARGE METHOD, the combined key name is TAB FEECHARGEDETL.CHARGE METHOD. The Chinese attribute description of this field is determined to be a charging method – one-time charge. This Chinese attribute description is then converted to the dictionary value “OT” as the key value. Therefore, the initial data dictionary can be “TAB FEECHARGEDETL.CHARGE METHOD-OT”. It should be noted that specific characters can be set according to the needs of technical personnel. In this embodiment of the invention, the preferred symbol is “-”, but this is not specifically limited.

[0044] In an optional embodiment of the present invention, the supplementary operation is the conversion of the initial data dictionary in key-value pair format into a baseline version of the target data dictionary. Specifically, the initial data dictionary is split based on specific characters, and a template for the baseline version is defined, including data table names, data field names, Chinese descriptions of data fields, and data field attribute values. The initial data dictionary in key-value pair format is converted into a baseline version data dictionary with a pre-configured template, and then presented to technical personnel. The technical personnel determine whether the initial data dictionary needs to be reviewed and supplemented. For example, if there are two charging methods: one-time charging and fixed-frequency charging, then after the system data is upgraded and optimized, if a new charging method is introduced, the new charging method is added. If the charging method is changed, the redundant charging methods can be deleted. The baseline version data dictionary reviewed and supplemented by the technical personnel is used as the target data dictionary.

[0045] Understandably, by automatically identifying and generating a data dictionary, the existing configuration-based rules are avoided from performing checks and scans. When a rule fails to run, the monitoring rule will no longer be executed, further reducing manual rule configuration and improving the efficiency and accuracy of subsequent system data detection.

[0046] S120. Perform syntactic analysis on the system data and construct a data link set by combining the path search algorithm.

[0047] Among them, syntax analysis involves parsing the data tables and data fields of the system data to obtain the processing relationships between the data. A syntax parser generator can be used to generate a syntax tree from text such as program code to obtain the processing relationships between the data; the path search algorithm is used to mine and simplify the node paths; and the data link set is the processing link corresponding to each data field.

[0048] Specifically, the system performs syntactic analysis on each data table to obtain the processing relationships between data fields. Combined with a path search algorithm, it mines, merges, and simplifies the paths of each node to determine the lineage paths between data tables and data fields. Based on the lineage paths, the processing relationships between data fields are converted into association relationships between data tables as processing links for each data field. The processing links for each data field are then integrated into a set of data links.

[0049] S130. Real-time detection of the system database based on the target data dictionary and data link set.

[0050] Specifically, the target data dictionary and data link set are imported into the system database, and the contents of table fields with the same key name in the database are checked periodically to determine whether there is data that has not been registered in the data dictionary. If so, it is abnormal data, and the abnormal data link corresponding to the abnormal data is found from the data link set.

[0051] Optionally, real-time monitoring of the system database can be performed based on the target data dictionary and data link set, including:

[0052] The system database is scanned in real time using the target data dictionary to obtain the scan results;

[0053] Identify the abnormal data fields based on the scan results;

[0054] Identify the abnormal data links corresponding to the abnormal data fields from the data link set; the abnormal data links include the impact links and the source tracing links corresponding to the abnormal data fields.

[0055] Among them, the abnormal data field is a data field that is not registered in the target data dictionary; the source tracing link is the processing link that generated the abnormal data field; and the impact link is the link corresponding to the data field that was generated by the abnormal data field or was affected by the abnormal data field.

[0056] Specifically, based on the target data dictionary, the table field content with the same key name in the system database is scanned to determine whether there is data that is not registered in the data dictionary. If the scan result shows that there is data, the searched data field is regarded as an abnormal data field. The processing link that generates the abnormal data field and the corresponding link of the data field generated by the abnormal data field or affected by the abnormal data field are automatically generated from the data link set.

[0057] Understandably, abnormal data is obtained through the target data dictionary, and abnormal impact analysis and cause analysis are performed on the abnormal data based on the data link set. The data processing context is fully tracked according to the source link and impact link to find all related data objects and relationships of a certain data object, assisting relevant personnel in quickly locating abnormal problems and the scope of their impact.

[0058] Optionally, after determining the abnormal data link corresponding to the abnormal data field from the data link set, the method further includes:

[0059] A graph algorithm is used to process each data table and data field in the abnormal data link to obtain an abnormal data graph structure; where each node in the abnormal data graph structure represents a data table or data field, and the edge represents the processing direction of the abnormal data link.

[0060] In response to the correction operation on the abnormal data graph structure, the abnormal feedback result is obtained;

[0061] The data link set and target data dictionary are corrected based on the anomaly feedback results.

[0062] Among them, graph algorithms can construct graph structures using graph structure libraries or in-memory graph libraries; the processing direction of abnormal data links is the upstream and downstream processing direction of abnormal data; the correction operation is the correction operation of the abnormal data graph structure by technicians based on practical experience; the abnormal feedback result is whether the abnormal judgment is correct or incorrect.

[0063] Specifically, based on graph algorithms, each data table and data field in the abnormal data link is processed to obtain an abnormal data graph structure, which is then displayed to relevant technical personnel. In this abnormal data graph structure, each node represents a data table or data field, and the edges connecting the nodes indicate the processing direction of the abnormal data link. Relevant technical personnel can make corrections and judgments based on their experience and obtain abnormal feedback results. If the abnormal feedback result indicates that the display is correct, data repair is performed based on the abnormal data link of the abnormal data. If the abnormal feedback result indicates that the display is abnormal, it means that the data is not abnormal, and the abnormal data field and the data table to which the abnormal data field belongs are added to the target data dictionary to optimize and update the target data dictionary.

[0064] Understandably, visualizing abnormal data links allows technical personnel to promptly assess the impact of the abnormal data and optimize or repair programs or data based on the feedback, or to optimize and update the target data dictionary.

[0065] In an optional embodiment of the present invention, the abnormal data graph structure can be visualized. Specifically, in the abnormal data link, each data table in the data link is displayed according to a preset display rule. The preset display rule maps the position of the data table in the data link to its color. It is known that the data link may include a source data table, an intermediate data table, and a result data table. Green represents the source data table, yellow represents the intermediate data table, and pink represents the result data table. That is, for the source tracing link: since the abnormal data field is obtained by processing the upstream data field, there can be multiple upstream data fields, for example: abnormal Data field g is the result data field. The abnormal data field g is obtained by processing data field f in table F, and data field f in table F is obtained by data field e in table E. Table E is a data table without input. In the tracing link of abnormal data field g, the data table G to which abnormal data field g belongs is the result data table; the data table F to which data field f belongs is the intermediate data table. The data table E without input is the source data table. The data graph structure of the source data table E, intermediate data table F and result data table G is color-processed, and the data link after color processing is displayed.

[0066] Regarding the influence chain, i.e. the data table chain resulting from the influence or processing of data fields: Taking data table B, to which the abnormal data field b belongs, as the source data table, processing the abnormal data field b yields field c, and data table C, to which field c belongs, is the intermediate data table; processing data field c yields data field d, determining that data table D belongs to data field d, and since data table D has no further processing, then data table D is the result data table. The value of the abnormal data field b influences field g in data table G, therefore it also belongs to the result data table in the influence chain. The data graph structure of source data table B, intermediate data table C, result data table D, and result data table G is color-coded, and the color-coded data chain is then displayed.

[0067] Understandably, visualizing the abnormal data graph structure allows technical personnel to quickly view the scope of the abnormal data's impact, accurately identify the cause of the anomaly, and conduct analysis. This ensures that technical personnel can promptly and quickly locate system anomalies, effectively improving the stability of system operation.

[0068] This invention traverses the system database to obtain system data. For each data table in the system data, it associates the table name with the data field names within the table to generate a target data dictionary. It then performs syntax analysis on the system data and constructs a data link set using a path search algorithm. Based on the target data dictionary and the data link set, it performs real-time monitoring of the system database. This technical solution solves the problem of quickly locating abnormal data. By scanning the database using the target data dictionary and data link set, if abnormal data is found, the abnormal data link is automatically exported, the anomaly is promptly registered, and the relevant maintenance personnel are notified. This achieves automatic monitoring of abnormal data and rapid location of the impact range, reducing the mean time to repair faults and improving system stability.

[0069] Figure 2 This is a flowchart of an abnormal data detection method according to an embodiment of the present invention. Based on the above embodiments, this embodiment supplements the construction method of the data link set. It should be noted that for parts not described in detail in this embodiment, please refer to the relevant descriptions in other embodiments, such as... Figure 2 As shown, the method includes:

[0070] S210. Traverse the system database to obtain system data. For each data table in the system data, associate the data table name with the data field names in the data table to generate the target data dictionary.

[0071] S220. Obtain the general workshop model corresponding to the system data; the general workshop model includes at least one data table.

[0072] Among them, the general workshop model is a framework for system data, used to describe the entire lifecycle flow of data from generation to consumption, and to record the input-output relationships between these processes.

[0073] Specifically, obtain the general workshop model corresponding to the system data. This general workshop model includes at least one data table and records the input and output relationships of the data fields in the data table.

[0074] S230. Perform syntax analysis on the data table to obtain at least two data fields and the processing relationship between the data fields; the processing relationship is the processing relationship between different data fields in the same data table or the processing relationship between different data fields in different data tables.

[0075] Among them, syntax analysis can be the parsing of code logic using ANTLR4 (Another Tool for Language Recognition); the processing relationship between different data fields in the same data table is the generation relationship between fields in the same data table; the processing relationship between different data fields in different data tables can be the processing relationship between data tables or the data field being processed by data fields in other data tables.

[0076] Specifically, for the data tables and corresponding data fields in the pre-built general workshop model, ANTLR4 can be used for syntax analysis to automatically parse the processing logic of the conversion code, extract the processing relationships between different data fields in the same data table or between different data fields in different data tables.

[0077] S240. Determine the lineage path of a data field using a path search algorithm; the lineage path includes the data table to which the data field belongs.

[0078] Among them, the lineage path is the relationship between each data field and the data table, that is, the data table to which the data field belongs.

[0079] Specifically, by using path search algorithms, node paths are mined, merged, and simplified to determine the lineage paths of data fields in the program code, i.e., the data tables to which each data field belongs.

[0080] S250. Construct a data link set based on the processing relationships and lineage paths between data fields.

[0081] Specifically, based on the processing relationships between data fields, the upstream and downstream data fields are identified, and the data tables to which the upstream and downstream data fields belong are determined based on the path information of the data fields. Based on the processing relationships between data fields, the corresponding upstream and downstream data tables are constructed as data links, and the data links corresponding to each data field are integrated into a data link set.

[0082] Optional, such as Figure 3 The method shown describes a method for determining a data link set, which constructs the data link set based on the processing relationships and lineage paths between data fields, including:

[0083] S251. For each data field, based on the processing relationship corresponding to the data field, determine the upstream data field and the downstream data field of the data field; the downstream data field includes the second data field obtained by processing the data field and / or the third data field affected by the data field.

[0084] Among them, the upstream data field is the data field from which the data field was processed; the second data field is the data field obtained by processing the data field; and the third data field is the data field affected by the value of the data field.

[0085] Specifically, for each data field, based on the processing relationship corresponding to the data field, that is, the context data fields that the data field processes and are processed, the data field that generates the data field is taken as the upstream data field, the data field obtained by processing the data field is taken as the second data field, the data field affected by the value of the data field is taken as the third data field, and the second data field and the third data field are taken as the downstream data field.

[0086] S252. Determine the upstream data table to which the upstream data field belongs and the downstream data table to which the downstream data field belongs from the lineage path.

[0087] Specifically, find the data table to which the data field belongs from the lineage path, that is, the upstream data table to which the upstream data field belongs, the second data table to which the second data field belongs in the downstream data field, and the third data table to which the third data field belongs. Then, use the second data table and the third data table as the downstream data table.

[0088] S253. Associate the upstream data table with the data table to which the data field belongs to construct the traceability link of the data field.

[0089] Specifically, the upstream data table is associated with the data table to which the data field belongs to obtain the traceability link corresponding to the data field, which indicates which data tables the data field is processed according to the process.

[0090] S254. Associate the downstream data table with the data table to which the data field belongs to construct the influence chain of the data field.

[0091] Specifically, the downstream data table is associated with the data table to which the data field belongs to obtain the influence chain corresponding to the data field, which indicates which data can be obtained by processing the data field, the data fields affected by the data field, and the data tables to which they belong.

[0092] S255. Construct a data link set based on the source link and impact link corresponding to each data field.

[0093] Specifically, the source tracing link and impact link corresponding to each data field are obtained as data links, and the data links are integrated to construct a data link set.

[0094] S260. Real-time detection of the system database based on the target data dictionary and data link set.

[0095] This invention performs syntactic analysis on system data to find the processing relationships between data fields, and then determines the influence and tracing links of each data field based on the lineage path. This ensures that after real-time detection of anomalies, the scope of impact and causes of the anomalies can be analyzed step by step, forming an analysis summary archive of the anomalies. This enables comprehensive tracking of the data processing context, finding all related data objects and relationships of a certain data object, assisting relevant personnel in quickly locating problems and the scope of impact, and improving the stability of system operation.

[0096] Figure 4 This is a schematic diagram of an abnormal data detection device according to an embodiment of the present invention. This embodiment is applicable to situations involving abnormal monitoring of system data in a system database. The abnormal data detection device can be implemented in hardware and / or software, and can be configured in a server. Figure 4 As shown, the abnormal data detection device 300 includes a target data dictionary generation module 310, a data link set construction module 320, and a detection module 330;

[0097] The target data dictionary generation module 310 is used to traverse the system database to obtain system data, and for each data table in the system data, associate the data table name and the data field name in the data table to generate the target data dictionary;

[0098] The data link set construction module 320 is used to perform syntactic analysis on system data and construct a data link set in combination with a path search algorithm;

[0099] The detection module 330 is used to perform real-time detection on the system database based on the target data dictionary and data link set.

[0100] This invention traverses the system database to obtain system data. For each data table in the system data, it associates the table name with the data field names within the table to generate a target data dictionary. It then performs syntax analysis on the system data and constructs a data link set using a path search algorithm. Based on the target data dictionary and the data link set, it performs real-time monitoring of the system database. This technical solution solves the problem of quickly locating abnormal data. By scanning the database using the target data dictionary and data link set, if abnormal data is found, the abnormal data link is automatically exported, the anomaly is promptly registered, and the relevant maintenance personnel are notified. This achieves automatic monitoring of abnormal data and rapid location of the impact range, reducing the mean time to repair faults and improving system stability.

[0101] Optionally, the detection module 330 is specifically used to perform real-time scanning of the system database through the target data dictionary to obtain the scanning results; determine the abnormal data fields based on the scanning results; determine the abnormal data links corresponding to the abnormal data fields from the data link set; the abnormal data links include the impact links and the source tracing links corresponding to the abnormal data fields.

[0102] Optionally, the abnormal data detection device 300 further includes a correction module, which is used to process each data table and data field in the abnormal data link using a graph algorithm to obtain an abnormal data graph structure; wherein, each node in the abnormal data graph structure represents a data table or data field, and the edge represents the processing direction of the abnormal data link; in response to the correction operation of the abnormal data graph structure, an abnormal feedback result is obtained; and the data link set and the target data dictionary are corrected based on the abnormal feedback result.

[0103] Optionally, the target data dictionary generation module 310 is specifically used to filter system data according to preset field thresholds to obtain target data; combine the data table name and field name to which the target data belongs as key names; use the attribute values ​​corresponding to the data table as key values; insert specific symbols between the key names and key values ​​to obtain an initial data dictionary; and obtain the target data dictionary in response to the supplementary operation of the initial data dictionary.

[0104] Optionally, the data link set construction module 320 includes a general workshop model acquisition unit, a syntax analysis unit, a lineage path determination unit, and a data link set construction unit;

[0105] A general workshop model acquisition unit is used to acquire the general workshop model corresponding to the system data; the general workshop model includes at least one data table.

[0106] The syntax analysis unit is used to perform syntax analysis on the data table to obtain at least two data fields and the processing relationships between the data fields; the processing relationships can be between different data fields in the same data table or between different data fields in different data tables.

[0107] The lineage path determination unit is used to determine the lineage path of a data field through a path search algorithm; the lineage path includes the data table to which the data field belongs;

[0108] The data link set construction unit is used to construct a data link set based on the processing relationships and lineage paths between data fields.

[0109] Optionally, the data link set construction unit is specifically used to, for each data field, determine the upstream and downstream data fields of the data field based on the processing relationship corresponding to the data field; the downstream data fields include the second data field obtained by processing the data field and / or the third data field affected by the data field; determine the upstream data table to which the upstream data field belongs and the downstream data table to which the downstream data field belongs from the lineage path respectively; associate the upstream data table with the data table to which the data field belongs to construct the source link of the data field; associate the downstream data table with the data table to which the data field belongs to construct the influence link of the data field; and construct a data link set based on the source link and influence link corresponding to each data field.

[0110] The abnormal data detection device provided in the embodiments of the present invention can execute the abnormal data detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0111] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0112] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0113] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0114] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as anomaly data detection methods.

[0115] In some embodiments, the abnormal data detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the abnormal data detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the abnormal data detection method by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0121] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0122] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An abnormal data detection method characterized by comprising: The method comprises the following steps: traversing a system database to obtain system data, associating a data table name of each data table in the system data with a data field name in the data table, and generating a target data dictionary; performing syntax analysis on the system data and constructing a data link set by combining a path search algorithm; performing real-time detection on the system database based on the target data dictionary and the data link set.

2. The method of claim 1, wherein, The real-time detection on the system database based on the target data dictionary and the data link set comprises: performing real-time scanning on the system database by using the target data dictionary to obtain a scanning result; determining an abnormal data field according to the scanning result; determining an abnormal data link corresponding to the abnormal data field from the data link set; the abnormal data link comprises an influence link and a traceability link corresponding to the abnormal data field.

3. The method of claim 2, wherein, After determining the abnormal data link corresponding to the abnormal data field from the data link set, the method further comprises: processing each data table and data field in the abnormal data link by using a graph algorithm to obtain an abnormal data graph structure; wherein each node in the abnormal data graph structure represents a data table or a data field, and an edge represents a processing direction of the abnormal data link; obtaining an abnormal feedback result in response to a modification operation on the abnormal data graph structure; modifying the data link set and the target data dictionary based on the abnormal feedback result.

4. The method of claim 1, wherein, The method of associating a data table name of each data table in the system data with a data field name in the data table to generate a target data dictionary comprises: filtering the system data according to a preset field threshold to obtain target data; combining a data table name and a field name to which the target data belongs as a key name; combining an attribute value corresponding to the data table as a key value; inserting a specific symbol between the key name and the key value to obtain an initial data dictionary; obtaining a target data dictionary in response to a supplement operation on the initial data dictionary.

5. The method of claim 1, wherein, The method of performing syntax analysis on the system data and constructing a data link set by combining a path search algorithm comprises: obtaining a general plant model corresponding to the system data; the general plant model comprises at least one data table; performing syntax analysis on the data table to obtain at least two data fields and a processing relationship between the data fields; the processing relationship is a processing relationship between different data fields in the same data table or a processing relationship between different data fields in different data tables; determining a blood relationship path of the data fields by using a path search algorithm; the blood relationship path comprises a data table to which the data field belongs; constructing a data link set according to the processing relationship between the data fields and the blood relationship path.

6. The method of claim 5, wherein, The method of constructing a data link set according to the processing relationship between the data fields and the blood relationship path comprises: for each data field, determining an upstream data field and a downstream data field of the data field based on a processing relationship corresponding to the data field; the downstream data field comprises a second data field processed by the data field and / or a third data field affected by the data field. determine an upstream data table to which the upstream data field belongs and a downstream data table to which the downstream data field belongs from the bloodline path respectively; associate the upstream data table with the data table to which the data field belongs, and construct a trace link of the data field; associate the downstream data table with the data table to which the data field belongs, and construct an influence link of the data field; construct a data link set based on the trace link and the influence link corresponding to each data field.

7. An abnormal data detection device characterized by comprising: comprise: a target data dictionary generation module, configured to traverse a system database to obtain system data, and for each data table in the system data, associate a data table name of the data table with a data field name in the data table, and generate a target data dictionary; a data link set construction module, configured to perform syntax analysis on the system data, and construct a data link set in combination with a path search algorithm; a detection module, configured to perform real-time detection on the system database based on the target data dictionary and the data link set.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the abnormal data detection method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the abnormal data detection method in any one of claims 1-6 when executed.

10. A computer program product, characterised in that, The computer program product comprises a computer program that, when executed by a processor, implements the abnormal data detection method according to any one of claims 1-6. The computer program product comprises a computer program that, when executed by a processor, implements the abnormal data detection method according to any one of claims 1-6.

Images