Metadata exception influence analysis method, device and equipment

By defining the graph data model and using ISM analysis methods, combined with the random forest algorithm, the problem of difficult to evaluate the scope and degree of impact in metadata anomalies is solved, and accurate monitoring and analysis of the impact of metadata anomalies is realized, and data management needs are met. .

CN119066588APending Publication Date: 2024-12-03GUANGZHOU JOYSLIM NETWORK TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411238457.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing technology fails to effectively analyze the scope and degree of influence of metadata exceptions, resulting in large losses caused by abnormal data and is difficult to meet the actual data management needs.

Method used

By defining the graph data model of target metadata, the flow path, location, dependency and real-time active value are extracted, structured influencing factors are generated using ISM analysis method, and the impact weight is calculated through the ANP network structure, and the data blood relationship is combined with the random forest algorithm. Impact prediction, monitor abnormal impact in real time.

Benefits of technology

It realizes accurate monitoring and analysis of the impact of metadata anomalies, reduces storage space usage, improves data access efficiency, ensures the accuracy and reliability of the impact coefficients, and meets the actual data management needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066588A_ABST
    Figure CN119066588A_ABST
Patent Text Reader

Abstract

The invention discloses a metadata exception influence analysis method, device and equipment. The method comprises the steps that a graph data model of target metadata is defined; extracting a flow path, a position, a dependency relationship and a real-time active value of the target metadata based on the mapped graph data set model, and analyzing influence factors according to an ISM analysis method to generate structured influence factors; calculating the influence weight of the structured influence factor in a mode of constructing an ANP network structure to obtain a weighted hypermatrix; performing data blood relationship influence prediction according to the structured influence factor and the influence weight vector by adopting a random forest algorithm to obtain a prediction influence coefficient; and monitoring the abnormal influence of the target metadata in real time according to the prediction influence coefficient to obtain a monitoring result. The technical problems that in the prior art, the influence of generated abnormal data is ignored, analysis of metadata lacks pertinence, the data analysis effect is poor, and actual data management requirements are difficult to meet can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a method, device and equipment for analyzing the impact of metadata anomaly. Background Art

[0002] With the advent of the digital age, data has become a new resource. The quality of metadata is crucial to the reliability and operability of data, and data lineage has become an important link in connecting metadata management, data governance and data quality.

[0003] When data anomalies occur, the cause of the anomaly can be traced through data lineage relationships, which helps to quickly locate the problem and take appropriate solutions. However, tracing data source only solves the source problem of the anomaly to prevent it from happening again, and cannot clearly understand the scope and extent of the impact caused by the anomaly that has occurred, so as to take corresponding remedial measures.

[0004] By observing the data lineage, or counting the number of data updates or the amount of data per unit time, we can evaluate the magnitude and frequency of data updates. The larger the magnitude or the more frequent the metadata updates, the greater the data influence. If the data quality is low, the probability of anomalies will increase. If the metadata with higher value has anomalies, the impact will be more serious. The existing technology does not consider the above issues, resulting in large losses caused by abnormal data and cannot meet the actual data management needs. Summary of the invention

[0005] The present application provides a metadata anomaly impact analysis method, device and equipment to solve the technical problems that the prior art ignores the impact of generated abnormal data and lacks pertinence in the analysis of metadata, resulting in poor data analysis results and difficulty in meeting actual data management needs.

[0006] In view of this, the first aspect of the present application provides a metadata anomaly impact analysis method, comprising:

[0007] Defining a graph data model of target metadata, the graph data model comprising graph nodes, relationships, labels, and attributes;

[0008] Extract the flow path, location, dependency and real-time active value of the target metadata based on the mapped graph dataset model, and perform influencing factor analysis according to the ISM analysis method to generate structured influencing factors;

[0009] Calculating the influence weights of the structured influence factors by constructing an ANP network structure to obtain a weighted supermatrix, wherein the weighted supermatrix includes a plurality of influence weight vectors;

[0010] Using a random forest algorithm, predicting the impact of data kinship based on the structured influencing factors and the influencing weight vector, and obtaining a predicted influencing coefficient;

[0011] The abnormal impact of the target metadata is monitored in real time according to the predicted impact coefficient to obtain a monitoring result.

[0012] Preferably, the graph dataset model after mapping processing extracts the flow path, location, dependency and real-time active value of the target metadata, and performs influencing factor analysis according to an ISM analysis method to generate structured influencing factors, including:

[0013] Mapping a preset multimodal data source to the graph data model through a relational database model to obtain a target metadata model;

[0014] Extracting the flow path, location, dependency, and real-time active value of the target metadata based on the target metadata model, and recording influencing factors;

[0015] Constructing an adjacency matrix for the influencing factors according to the ISM analysis method, and performing hierarchical relationship impact analysis based on the adjacency matrix to obtain a hierarchical structure of the influencing factors;

[0016] The influencing factors and the influencing factor hierarchical structure are combined to generate structured influencing factors.

[0017] Preferably, mapping the preset multimodal data source to the graph data model through a relational database model to obtain a target metadata model includes:

[0018] Access pre-set multimodal data sources through metadata management tools;

[0019] Mapping the tables in the preset multimodal data source to entities in the graph data model through the relational database model in the metadata management tool to obtain entity nodes and node labels;

[0020] Mapping columns in the preset multimodal data source to node attributes in the graph data model;

[0021] Mapping foreign keys in the preset multimodal data source to relationship edges in the graph data model;

[0022] A target metadata model is determined based on the entity nodes, the node labels, the node attributes, and the relationship edges.

[0023] Preferably, mapping the columns in the preset multimodal data source to node attributes in the graph data model further includes:

[0024] The attribute set of each entity node is optimized and sorted by deletion and merging to obtain optimized node attributes.

[0025] Preferably, extracting the flow path, location, dependency and real-time active value of the target metadata based on the target metadata model and recording influencing factors include:

[0026] Determine the source and type of the target metadata based on the target metadata model, and record the flow path of the target metadata in the system based on the source and type;

[0027] Analyze the location of the target metadata and the dependency relationship between data according to the directory changes of the target metadata;

[0028] Monitor the change dynamics of the target metadata in real time, and determine the real-time active value of the target metadata according to the change dynamics through a preset statistical method;

[0029] The influencing factors of the target metadata are recorded according to the flow path, the location, the dependency, and the real-time active value.

[0030] Preferably, the abnormal impact of the target metadata is monitored in real time according to the predicted impact coefficient to obtain a monitoring result, and then the method further includes:

[0031] The monitoring results are presented through visualization tools to form a metadata lineage relationship report.

[0032] A second aspect of the present application provides a metadata anomaly impact analysis device, comprising:

[0033] A model definition unit, used to define a graph data model of target metadata, wherein the graph data model includes graph nodes, relationships, labels, and attributes;

[0034] An impact analysis unit, used to extract the flow path, location, dependency and real-time active value of the target metadata based on the graph data set model after mapping, and perform impact factor analysis according to an ISM analysis method to generate structured impact factors;

[0035] A weight calculation unit, used for calculating the influence weight of the structured influence factor by constructing an ANP network structure to obtain a weighted supermatrix, wherein the weighted supermatrix includes a plurality of influence weight vectors;

[0036] A coefficient prediction unit, configured to use a random forest algorithm to predict the influence of data kinship according to the structured influence factors and the influence weight vector, and obtain a predicted influence coefficient;

[0037] The impact monitoring unit is used to monitor the abnormal impact of the target metadata in real time according to the predicted impact coefficient to obtain a monitoring result.

[0038] Preferably, the impact analysis unit comprises:

[0039] A data mapping subunit, used to map a preset multimodal data source to the graph data model through a relational database model to obtain a target metadata model;

[0040] A factor extraction subunit, for extracting the flow path, location, dependency and real-time active value of the target metadata based on the target metadata model, and recording influencing factors;

[0041] An impact analysis subunit is used to construct an adjacency matrix for the impact factors according to an ISM analysis method, and to perform hierarchical impact analysis based on the adjacency matrix to obtain a hierarchical structure of the impact factors;

[0042] The structure generation subunit is used to generate structured influencing factors by combining the influencing factors and the influencing factor hierarchical structure.

[0043] Preferably, the data mapping subunit is specifically used for:

[0044] Access pre-set multimodal data sources through metadata management tools;

[0045] Mapping the tables in the preset multimodal data source to entities in the graph data model through the relational database model in the metadata management tool to obtain entity nodes and node labels;

[0046] Mapping columns in the preset multimodal data source to node attributes in the graph data model;

[0047] Mapping foreign keys in the preset multimodal data source to relationship edges in the graph data model;

[0048] A target metadata model is determined based on the entity nodes, the node labels, the node attributes, and the relationship edges.

[0049] A third aspect of the present application provides a metadata anomaly impact analysis device, the device comprising a processor and a memory;

[0050] The memory is used to store program code and transmit the program code to the processor;

[0051] The processor is used to execute the metadata anomaly impact analysis method described in the first aspect according to the instructions in the program code.

[0052] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0053] In the present application, a method for analyzing the impact of abnormal metadata is provided, comprising: defining a graph data model of target metadata, the graph data model comprising graph nodes, relationships, labels and attributes; extracting the flow path, location, dependency and real-time active value of the target metadata based on the graph data set model after mapping processing, and performing influencing factor analysis according to the ISM analysis method to generate structured influencing factors; calculating the influence weights of the structured influencing factors by constructing an ANP network structure to obtain a weighted supermatrix, the weighted supermatrix comprising multiple influence weight vectors; using a random forest algorithm to predict the impact of data lineage relationships according to the structured influencing factors and the influence weight vectors to obtain a predicted influence coefficient; and performing real-time monitoring of the abnormal impact of the target metadata according to the predicted influence coefficient to obtain a monitoring result.

[0054] The metadata abnormal impact analysis method provided by the present application defines the target metadata as a graph data model to reduce the storage space occupied, and can quickly access information such as data relationships and attributes by virtue of the association relationship between graph nodes; and timely extracting the basic information and flow change information of the metadata and analyzing the impact of factors based on the graph data model can not only realize the traceability of metadata, but also analyze its impact on upstream and downstream blood relationship data based on the influencing factors, so as to obtain a quantitative indicator through the random forest algorithm, that is, the predicted impact coefficient; this process specifically considers the data value of metadata activity and real-time, and can ensure the accuracy and reliability of the obtained impact coefficient, so the abnormal impact monitoring based on this coefficient is also reliable. Therefore, the present application can solve the technical problems that the prior art ignores the impact of the abnormal data that has been generated, and the analysis of metadata lacks pertinence, resulting in poor data analysis results and difficulty in meeting actual data management needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a metadata anomaly impact analysis method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the structure of a metadata anomaly impact analysis device provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the elements of the graph data model provided in the embodiment of the present application;

[0058] Figure 4 This is an example diagram of a hierarchical structured influencing factor process generated according to the ISM analysis method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] For easier understanding, see Figure 1 , an embodiment of a metadata anomaly impact analysis method provided by the present application includes:

[0061] Step 101: Define a graph data model of target metadata, where the graph data model includes graph nodes, relationships, labels, and attributes.

[0062] See also Figure 3 In this embodiment, the target metadata is defined in the form of a graph data model, which essentially stores the metadata in the form of nodes, relationships, labels, and attributes; wherein the relationships can describe the associations between nodes, and the attributes contain the relevant description information of the nodes.

[0063] At present, most relational databases are used to store metadata. In order to ensure the consistency and integrity of data, relational databases need to store redundant data, which increases the proportion of data storage space and reduces the speed of data processing, which is inefficient. In this embodiment, the metadata stored based on the graph data model describes data entities based on graph nodes. The nodes are connected in various ways, which can ensure quick access to the associations between various nodes and edges. The graph data model has high modeling flexibility and does not rely on fixed data models. It is more suitable for processing complex and changeable metadata sets.

[0064] Step 102, based on the mapped graph dataset model, the flow path, location, dependency, and real-time active value of the target metadata are extracted, and the influencing factor analysis is performed according to the ISM analysis method to generate structured influencing factors.

[0065] Furthermore, step 102 includes:

[0066] S1: Map the preset multimodal data source to the graph data model through the relational database model to obtain the target metadata model.

[0067] Furthermore, step S1 includes:

[0068] Access pre-set multimodal data sources through metadata management tools;

[0069] Map the tables in the preset multimodal data source to entities in the graph data model through the relational database model in the metadata management tool to obtain entity nodes and node labels;

[0070] Map the columns in the preset multimodal data source to node attributes in the graph data model;

[0071] Map foreign keys in the pre-set multimodal data source to relationship edges in the graph data model;

[0072] The target metadata model is determined based on entity nodes, node labels, node attributes, and relationship edges.

[0073] Furthermore, mapping the columns in the preset multimodal data source to node attributes in the graph data model also includes:

[0074] The attribute set of each entity node is optimized and sorted by deletion and merging to obtain optimized node attributes.

[0075] It should be noted that the pre-set multimodal data sources accessed through the metadata management tool can be mapped to the nodes and edges of the graph data model through cleaning, conversion and formatting to generate data lineage relationships. The relational database model in the metadata management tool can serve as a mapping bridge to achieve information mapping and obtain the target metadata model.

[0076] Specifically, the graph data model uses nodes to represent entities, while in the relational database model, each entity is represented by a table. Therefore, the tables in the preset multimodal data source can be regarded as entities mapped to the graph data model to form entity nodes, and the table names can be mapped to the corresponding node labels. Each row in the table represents an entity instance, and an entity instance can be mapped to a separate node, that is, an entity node or a graph node.

[0077] The columns or fields in the table represent the attributes of the entity, which are mapped to the graph data model as the attributes of the corresponding nodes. In other words, the attribute set of each entity node is the column value of the corresponding row in its original relational table.

[0078] The foreign keys in the preset multimodal data source are used to represent the relationship between tables. When mapped to the graph data model, they can be represented by edges; that is, nodes are connected by edges to describe the connection relationship between entities. If the original foreign key column / field is not needed in the node, it can be deleted; if it is needed, it can be mapped as an edge attribute. Based on the entity nodes, node labels, node attributes, and relationship edges obtained from the above mapping operations, the target metadata model can be determined.

[0079] In addition, in order to optimize node attributes and reduce the complexity of metadata storage, the present embodiment can also sort and optimize the attributes of nodes or edges during the mapping process. Unnecessary columns or attributes can be removed, and the same or similar columns can be merged. Specifically, the technical primary key can be deleted and the business primary key can be retained. If certain default values ​​are not important in the business logic, you can choose to omit them in the diagram. The index column of an array, for example, the index column name may represent an array attribute; in a relational database model, multiple columns are sometimes used to store data that should be an array; during the mapping process, these columns are usually merged into an array attribute, which can contain any number of image addresses.

[0080] In addition, a unique constraint can be added to the business primary key to ensure that there are no duplicate nodes in the graph. Indexes can also be added to frequently searched attributes to improve query efficiency. Furthermore, duplicate data in the non-normalized table must be pulled into separate nodes to obtain a clear model. It can be analyzed that the graph information mapping method of this embodiment is more efficient than the prior art because it simplifies the process of building data kinship relationships.

[0081] S2: Extract the flow path, location, dependency, and real-time active value of the target metadata based on the target metadata model, and record the influencing factors.

[0082] Further, step S2 includes:

[0083] Determine the source and type of target metadata based on the target metadata model, and record the flow path of target metadata in the system based on the source and type;

[0084] Analyze the location of target metadata and the dependencies between data according to the directory changes of target metadata;

[0085] Monitor the changes of target metadata in real time, and determine the real-time active value of target metadata based on the changes through preset statistical methods;

[0086] Record the influencing factors of target metadata based on flow paths, locations, dependencies, and real-time active values.

[0087] It should be noted that the source and type of the target metadata that needs to be analyzed can be directly determined in the target metadata model, and the flow process of data in the system can be recorded, and the change history and conversion logic of the data can be tracked. Therefore, this embodiment can determine the flow path, location and dependencies between data of the metadata based on the target metadata model. Moreover, at different stages, the location and dependencies of the metadata directory can be analyzed regularly according to the changes in the metadata directory. These are all information that can be changed with the stage changes, and this information can reflect the impact and expansion of the metadata to a certain extent.

[0088] Because metadata will constantly change and transform as it flows, it is necessary to capture metadata changes in real time and analyze the changes to determine the activity and real-time nature of metadata. The preset statistical method refers to evaluating the value of metadata by counting the number of data changes or the amount of data per unit time; based on statistics, real-time active value can be generated. In addition, dynamic monitoring of changes can be achieved by configuring inspection jobs, running jobs, viewing job execution logs, viewing error data and inspection results, etc.; and real-time can be regular timing, and the specific time interval is not limited here.

[0089] The impact of metadata on external scalability can be analyzed based on flow paths, locations, dependencies, and real-time active values, and the influencing factors of metadata can be analyzed and recorded for subsequent abnormal impact analysis.

[0090] Since the extracted information is highly targeted, the extracted influencing factors are representative and can reflect the impact of metadata. Moreover, current technology does not consider the impact of metadata’s own value on data quality, and its anomaly identification or alarm is not accurate enough.

[0091] S3: Construct an adjacency matrix for the influencing factors based on the ISM analysis method, and perform hierarchical relationship impact analysis based on the adjacency matrix to obtain the hierarchical structure of the influencing factors.

[0092] S4: Combine influencing factors and the hierarchical structure of influencing factors to generate structured influencing factors.

[0093] It should be noted that the adjacency matrix of influencing factors can be constructed according to the ISM analysis method. If the set of influencing factors is expressed as , then the adjacency Boolean matrix expressing the relevant influence relationship between any two influencing factor nodes can be expressed as , that is, the adjacency matrix, then we have:

[0094]

[0095] For the specific operation process, please refer to Figure 4 :According to the adjacency Boolean matrix A, the adjacency multiplication matrix B=A×A can be calculated. The shrunken point matrix after all strongly connected subsets obtained by the Tarjan algorithm is the lower triangular rearranged shrunken point matrix L. Using topological operations, the skeleton matrix S' is obtained, that is, the matrix obtained after L is shrunk. The general skeleton matrix S is obtained, and the loop is represented by a daisy chain connected end to end. When there is no loop, S=S'. The UP|DOWN hierarchical partitioning rule is based on the general skeleton matrix S plus the unit matrix I, that is, the main diagonal is filled with 1:

[0096] The corresponding reachable set ,Right now The feature corresponds to all features whose row value is 1.

[0097] Corresponding reason set ,Right now All features whose feature corresponding column value is 1.

[0098] The corresponding common set ,Right now = .

[0099] According to the hierarchical division of result priority, the extraction rules are as follows: ;

[0100] According to the hierarchy of cause priority, the extraction rules are as follows: .

[0101] By determining the influencing factor hierarchy based on the association and extraction results of various influencing factors, we can obtain an influencing factor hierarchy model, which is expressed as an adversarial hierarchical topological structure. This model is composed of structured influencing factors.

[0102] Step 103: Calculate the influence weights of the structured influence factors by constructing an ANP network structure to obtain a weighted supermatrix, where the weighted supermatrix includes multiple influence weight vectors.

[0103] It should be noted that based on the structured influencing factors obtained above, an ANP network structure including a control layer and a network layer can be constructed, and based on this, the degree of relevant influence between various influencing factors can be determined, which is specifically expressed by weights to obtain a weighted supermatrix. The target weight of each influencing factor can be obtained through extreme sorting to form an influence weight vector.

[0104] Specifically, according to the simplified topological diagram of the structured influencing factors obtained by ISM above, the control layer and network layer of the ANP network structure are determined. Among them, the control layer is the influencing factor set , that is, all structural influencing factors; the network layer includes tuples ,in It also includes multiple elements, expressed as ,in, ; That is, the network structure composed of the influencing factors controlled by the control layer.

[0105] Tuple As a guideline, middle The elements are sub-criteria, and the 1-9 table scaling method is introduced to compare the criteria and sub-criteria in pairs to determine their relative importance or priority, and construct the judgment matrix R:

[0106]

[0107] After consistency check, the influence weight vector is calculated by arithmetic mean method. :

[0108]

[0109] Combine the pairwise comparison criteria and the weight vectors of the sub-criteria into a supermatrix W:

[0110]

[0111] For each tuple under the control layer (i = 1,2, … N) for the subcriteria ( ) is compared with the importance of the weighted matrix through consistency test :

[0112]

[0113] Supermatrix By weighting the elements of , whose columns sum to 1, and .Will Normalize it and get the limit supermatrix ,exist When:

[0114]

[0115] in, for The tth power of The jth column of is the relative ranking weight vector of the elements of the network layer for element j under a certain decision criterion:

[0116]

[0117] This is the influence weight vector required in this embodiment.

[0118] Step 104: Use the random forest algorithm to predict the impact of data kinship based on the structured influencing factors and the influencing weight vector to obtain a predicted impact coefficient.

[0119] It should be noted that the random forest algorithm has good performance and generalization ability; by taking the entity nodes and relationship edges of the target metadata model, that is, the influencing total weight vector as the algorithm model features and input, and performing metadata impact analysis according to different structured influencing factors, the corresponding predicted impact coefficient of the output can be calculated.

[0120] This embodiment introduces the influence coefficient of metadata quality, and analyzes the position of metadata in the graph and the dependencies between metadata by tracing the flow path of data, and analyzes the data value of metadata by analyzing the activity and real-time performance of the statistical data source, so as to identify metadata anomalies more accurately and comprehensively.

[0121] It is understandable that the influence coefficient prediction process based on the random forest algorithm requires the construction of a corresponding prediction model, and then optimization training is performed based on the corresponding training set and test set, and the optimization model is selected as the target prediction model. The model is evaluated using the test data set, and the final prediction value can be obtained by averaging the prediction results of all decision trees; the specific operation process is not described in detail.

[0122] Step 105: Perform real-time monitoring on the abnormal impact of the target metadata based on the predicted impact coefficient to obtain a monitoring result.

[0123] The predicted impact coefficient can reflect the impact of metadata on upstream and downstream data. Therefore, real-time monitoring of abnormal impacts on metadata based on the predicted impact coefficient can ensure the accuracy and reliability of the monitoring results and meet the current data management needs. Based on the degree of impact reflected by the predicted impact coefficient, timely countermeasures can be taken to improve data quality.

[0124] Furthermore, step 105 further includes:

[0125] The monitoring results are displayed through visualization tools to form a metadata lineage report.

[0126] In order to facilitate data analysis and intuitively reflect the clear data lineage, this embodiment can also visualize the constructed target metadata model, predicted impact coefficient and monitoring results through visualization tools. The predicted impact coefficient can be expressed by the size of the entity node or the attribute value to show the impact of the metadata anomaly; the larger the bubble / value, the greater the data value; if the anomaly causes a large impact, it can be more intuitive and quick to conduct anomaly investigation and emergency processing, which can improve the user experience to a certain extent.

[0127] It can be understood that the metadata lineage report formed may include but is not limited to data lineage, lineage analysis results, impact analysis results and data quality, and these results can be obtained by data analysis based on the association relationship between the nodes in the graph, and the details will not be repeated here.

[0128] The metadata abnormal impact analysis method provided in the embodiment of the present application defines the target metadata as a graph data model to reduce the storage space occupied, and can quickly access information such as data relationships and attributes by virtue of the association relationship between graph nodes; and timely extracting the basic information and flow change information of the metadata and analyzing the impact of factors based on the graph data model can not only realize the traceability of metadata, but also analyze its impact on upstream and downstream blood relationship data based on the influencing factors, so as to obtain a quantitative indicator through the random forest algorithm, that is, the predicted impact coefficient; this process specifically considers the data value of metadata activity and real-time, and can ensure the accuracy and reliability of the obtained impact coefficient, so the abnormal impact monitoring based on this coefficient is also reliable. Therefore, the embodiment of the present application can solve the technical problems that the prior art ignores the impact of the generated abnormal data, and the analysis of metadata lacks pertinence, resulting in poor data analysis effect and difficulty in meeting actual data management needs.

[0129] For easier understanding, see Figure 2 The present application provides an embodiment of a metadata anomaly impact analysis device, including:

[0130] A model definition unit 201 is used to define a graph data model of target metadata, wherein the graph data model includes graph nodes, relationships, labels and attributes;

[0131] The impact analysis unit 202 is used to extract the flow path, location, dependency and real-time active value of the target metadata based on the mapped graph data set model, and perform impact factor analysis according to the ISM analysis method to generate structured impact factors;

[0132] The weight calculation unit 203 is used to calculate the influence weight of the structured influence factor by constructing an ANP network structure to obtain a weighted super matrix, wherein the weighted super matrix includes a plurality of influence weight vectors;

[0133] The coefficient prediction unit 204 is used to use a random forest algorithm to predict the influence of data blood relationship according to the structured influence factors and the influence weight vector to obtain a predicted influence coefficient;

[0134] The impact monitoring unit 205 is used to monitor the abnormal impact of the target metadata in real time according to the predicted impact coefficient to obtain a monitoring result.

[0135] Furthermore, the impact analysis unit 202 includes:

[0136] The data mapping subunit 2021 is used to map the preset multimodal data source to the graph data model through the relational database model to obtain a target metadata model;

[0137] The factor extraction subunit 2022 is used to extract the flow path, location, dependency and real-time active value of the target metadata based on the target metadata model, and record the influencing factors;

[0138] The impact analysis subunit 2023 is used to construct an adjacency matrix for the impact factors according to the ISM analysis method, and perform hierarchical relationship impact analysis based on the adjacency matrix to obtain a hierarchical structure of the impact factors;

[0139] The structure generation subunit 2024 is used to generate structured influencing factors by combining the influencing factors and the hierarchical structure of the influencing factors.

[0140] Furthermore, the data mapping subunit 2021 is specifically used for:

[0141] Access pre-set multimodal data sources through metadata management tools;

[0142] Map the tables in the preset multimodal data source to entities in the graph data model through the relational database model in the metadata management tool to obtain entity nodes and node labels;

[0143] Map the columns in the preset multimodal data source to node attributes in the graph data model;

[0144] Map foreign keys in the pre-set multimodal data source to relationship edges in the graph data model;

[0145] The target metadata model is determined based on entity nodes, node labels, node attributes, and relationship edges.

[0146] The present application also provides a metadata anomaly impact analysis device, the device comprising a processor and a memory;

[0147] The memory is used to store the program code and transmit the program code to the processor;

[0148] The processor is used to execute the metadata anomaly impact analysis method in the above method embodiment according to the instructions in the program code.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0153] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for analyzing the impact of metadata anomalies, It is characterized in that include: Defining a graph data model of target metadata, the graph data model comprising graph nodes, relationships, labels, and attributes; Extract the flow path, location, dependency and real-time active value of the target metadata based on the mapped graph dataset model, and perform influencing factor analysis according to the ISM analysis method to generate structured influencing factors; Calculating the influence weights of the structured influence factors by constructing an ANP network structure to obtain a weighted supermatrix, wherein the weighted supermatrix includes a plurality of influence weight vectors; Using a random forest algorithm, predicting the impact of data kinship based on the structured influencing factors and the influencing weight vector, and obtaining a predicted influencing coefficient; The abnormal impact of the target metadata is monitored in real time according to the predicted impact coefficient to obtain a monitoring result.

2. The metadata anomaly impact analysis method according to claim 1, It is characterized in that The graph dataset model after mapping processing extracts the flow path, location, dependency and real-time active value of the target metadata, and performs influencing factor analysis according to the ISM analysis method to generate structured influencing factors, including: Mapping a preset multimodal data source to the graph data model through a relational database model to obtain a target metadata model; Extracting the flow path, location, dependency, and real-time active value of the target metadata based on the target metadata model, and recording influencing factors; Constructing an adjacency matrix for the influencing factors according to the ISM analysis method, and performing hierarchical relationship impact analysis based on the adjacency matrix to obtain a hierarchical structure of the influencing factors; The influencing factors and the influencing factor hierarchical structure are combined to generate structured influencing factors.

3. The metadata anomaly impact analysis method according to claim 2, It is characterized in that The method of mapping the preset multimodal data source to the graph data model through the relational database model to obtain the target metadata model includes: Access pre-set multimodal data sources through metadata management tools; Mapping the tables in the preset multimodal data source to entities in the graph data model through the relational database model in the metadata management tool to obtain entity nodes and node labels; Mapping columns in the preset multimodal data source to node attributes in the graph data model; Mapping foreign keys in the preset multimodal data source to relationship edges in the graph data model; A target metadata model is determined based on the entity nodes, the node labels, the node attributes, and the relationship edges.

4. The metadata anomaly impact analysis method according to claim 3, It is characterized in that The mapping of the columns in the preset multimodal data source to node attributes in the graph data model further includes: The attribute set of each entity node is optimized and sorted by deletion and merging to obtain optimized node attributes.

5. The metadata anomaly impact analysis method according to claim 2, It is characterized in that The extracting of the flow path, location, dependency, and real-time active value of the target metadata based on the target metadata model, and recording influencing factors, includes: Determine the source and type of the target metadata based on the target metadata model, and record the flow path of the target metadata in the system based on the source and type; Analyze the location of the target metadata and the dependency relationship between data according to the directory changes of the target metadata; Monitor the change dynamics of the target metadata in real time, and determine the real-time active value of the target metadata according to the change dynamics through a preset statistical method; The influencing factors of the target metadata are recorded according to the flow path, the location, the dependency, and the real-time active value.

6. The metadata anomaly impact analysis method according to claim 1, It is characterized in that The abnormal impact of the target metadata is monitored in real time according to the predicted impact coefficient to obtain a monitoring result, and then further includes: The monitoring results are presented through a visualization tool to form a metadata lineage report.

7. A metadata anomaly impact analysis device, It is characterized in that include: A model definition unit, used to define a graph data model of target metadata, wherein the graph data model includes graph nodes, relationships, labels, and attributes; An impact analysis unit, used to extract the flow path, location, dependency and real-time active value of the target metadata based on the mapped graph data set model, and perform impact factor analysis according to an ISM analysis method to generate structured impact factors; A weight calculation unit, used for calculating the influence weight of the structured influence factor by constructing an ANP network structure to obtain a weighted supermatrix, wherein the weighted supermatrix includes a plurality of influence weight vectors; A coefficient prediction unit, configured to use a random forest algorithm to predict the influence of data kinship according to the structured influence factors and the influence weight vector, and obtain a predicted influence coefficient; The impact monitoring unit is used to monitor the abnormal impact of the target metadata in real time according to the predicted impact coefficient to obtain a monitoring result.

8. The metadata anomaly impact analysis device according to claim 7, It is characterized in that The impact analysis unit comprises: A data mapping subunit, used to map a preset multimodal data source to the graph data model through a relational database model to obtain a target metadata model; A factor extraction subunit, for extracting the flow path, location, dependency and real-time active value of the target metadata based on the target metadata model, and recording influencing factors; An impact analysis subunit is used to construct an adjacency matrix for the impact factors according to an ISM analysis method, and to perform hierarchical impact analysis based on the adjacency matrix to obtain a hierarchical structure of the impact factors; The structure generation subunit is used to generate structured influencing factors by combining the influencing factors and the influencing factor hierarchical structure.

9. The metadata anomaly impact analysis device according to claim 8, It is characterized in that The data mapping subunit is specifically used for: Access pre-set multimodal data sources through metadata management tools; Mapping the tables in the preset multimodal data source to entities in the graph data model through the relational database model in the metadata management tool to obtain entity nodes and node labels; Mapping columns in the preset multimodal data source to node attributes in the graph data model; Mapping foreign keys in the preset multimodal data source to relationship edges in the graph data model; A target metadata model is determined based on the entity nodes, the node labels, the node attributes, and the relationship edges.

10. A metadata anomaly impact analysis device, It is characterized in that The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the metadata anomaly impact analysis method according to any one of claims 1 to 6 according to the instructions in the program code.