An automated metadata management method, system, device, and storage medium

Abstract

The present invention discloses an automated metadata management method, system, device and storage medium, and relates to the technical fields of data governance and metadata management, including the system adapting different data sources, automatically connecting to a database through data source configuration, automatically collecting metadata matching the metamodel into a metadata database according to a metamodel and collection task configuration; defining a metamodel library to automatically collect current metadata information of a metamodel from a database, and verifying whether the collected metadata needs to be stored and upgraded; collecting data by associating tables and fields in a metadata database, using data development components and automatically collecting data from a data warehouse and developing after configuring filtering conditions, and automatically building a blood relationship. The present invention reduces manual intervention, improves the efficiency of metadata management, ensures the consistency and accuracy of metadata, establishes blood relationships of metadata tables, clearly displays the flow and dependency of data between different tables, and facilitates data management and tracing.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power engineering, and in particular relates to an automatic metadata management method, system, equipment and storage medium. Background Art

[0002] In modern power systems, the stable operation of steam turbine generator sets is the key to ensuring the continuity and reliability of power supply. However, in the long-term operation of the units, it is inevitable that the units will encounter outages caused by equipment failures, maintenance needs or external interference. According to the regulations of the power industry, the outages of generator sets are divided into two types: planned outages and unplanned outages. Planned outages are pre-arranged shutdown operations, usually used for equipment maintenance or system upgrades, aimed at ensuring the long-term safe and stable operation of the units. Unplanned outages are unexpected outages caused by sudden failures or other unexpected events, which will not only affect the stability of power supply, but may also pose a threat to the overall safety of the power system.

[0003] Traditional unit outage monitoring and management methods mainly rely on manual inspections and data recording. Although this method can meet basic management needs to a certain extent, with the complexity of power systems and the sharp increase in data volume, this manual monitoring method faces serious efficiency and accuracy issues.

[0004] A large amount of data needs to be collected and processed manually, which is not only time-consuming and labor-intensive, but also easily leads to data omissions or errors, which in turn affects the timely reporting and handling of outage events.

[0005] Due to reliance on manual reporting, information on unit failures cannot be uploaded in a timely manner, which may adversely affect scheduling and power balance and increase the pressure on grid operation.

[0006] Manual operation is prone to omissions or false alarms, which makes it impossible for regulatory authorities to accurately grasp the actual operating conditions of the unit, thus affecting decision-making and the implementation of subsequent measures.

[0007] In view of the existence of the above-mentioned problems, the present invention proposes a model-driven unit non-shutdown supervision method, which aims to overcome the shortcomings of traditional methods through the construction of intelligent models and automatic processing of real-time data, and realize accurate monitoring of the operating status of generator sets and automatic classification and reporting of shutdown events. Summary of the invention

[0008] In view of the above-mentioned problems, the present invention is proposed.

[0009] Therefore, the technical problem solved by the present invention is: the existing generator set shutdown monitoring method has the problems of low manual monitoring efficiency, delayed information feedback and insufficient data accuracy, as well as how to optimize the real-time monitoring and automatic classification of the generator set operating status.

[0010] To solve the above technical problems, the present invention provides the following technical solutions: an automated metadata management method, comprising: adapting the data source and connecting to the database, collecting metadata that matches the metamodel; defining the metamodel library, and collecting the current metadata information of the metamodel from the database; verifying whether the collected metadata needs to be stored and upgraded; automatically collecting data from the data warehouse for data development; obtaining user-specified indicator information to build a blood relationship, and predicting dependency relationships to locate abnormal data.

[0011] As a preferred solution of the automatic metadata management method described in the present invention, the metadata collected to match the meta-model includes:

[0012] Metadata collection includes data source definition, collection task configuration, collection task execution and metadata information storage;

[0013] The collection task configuration includes data source, metamodel classification, collection strategy, automatic storage, collection frequency and collection content;

[0014] After the collection task is released, a scheduled task is generated. When the specified time comes, the collection task starts automatically and collects and stores metadata according to the configured meta-model.

[0015] The collection content is that when clicking the add operation, the configured meta-model is selected from the meta-model library. After the meta-model is added, the collection task will collect the metadata information of the configured meta-model.

[0016] As a preferred solution of the automatic metadata management method described in the present invention, the step of verifying whether the collected metadata needs to be stored and updated includes:

[0017] When the collection task is started automatically, query the metadata collected before this collection task and obtain the metadata object;

[0018] Store the IDs of all metadata objects as keywords in the metadata collection map;

[0019] The metadata collection task obtains metadata information corresponding to the metamodel from the databaseMetaData object according to the metamodel;

[0020] The metadata information generates a metadata object, defines the id of the metadata object as the keyword, and queries whether the metadata collection map contains the keyword of the current metadata object;

[0021] When the map does not contain a keyword, the current metadata object does not exist in the metadata database, and data is inserted.

[0022] As a preferred solution of the automatic metadata management method described in the present invention, the verification of whether the collected metadata needs to be stored and upgraded also includes:

[0023] When the map contains keywords, it determines whether the current metadata object is equal to the metadata object in the metadata collection map;

[0024] If the current metadata object is equal to the metadata object in the metadata collection map, no processing is required;

[0025] If the current metadata object is not equal to the metadata object in the metadata collection map, the metadata object in the metadata database is recorded as a historical version, and the current metadata object is stored in the database as the latest version, and the version number is increased by 1.

[0026] As a preferred solution of the automatic metadata management method described in the present invention, the data development includes:

[0027] Data collection is performed through tables and fields in the associated metadata database;

[0028] Store all data in the metadata corresponding table into the data warehouse;

[0029] The data warehouse includes ODS, DWD, DWS, and ADS;

[0030] Determine the metadata collected in the system based on the data source;

[0031] The provided output component automatically writes the collected results to the ODS layer of the specified data warehouse;

[0032] Cleans and converts the data and automatically writes it to the DWD layer through the output component;

[0033] Perform preliminary aggregation and calculation based on user data and write it to the DWS layer of the specified data warehouse;

[0034] The data in the DWS layer is aggregated and calculated again at a high level and written into the ADS layer;

[0035] Automatically generate charts through chart components and display them on web pages.

[0036] As a preferred solution of the automatic metadata management method described in the present invention, wherein: the building of blood relationship includes:

[0037] In the process of data development in the data warehouse, establish the blood relationship between metadata tables and fields, use SpringAOP to intercept SQL execution statements, and push the intercepted SQL statements to the Kafka message queue to automatically identify data blood relationships;

[0038] The intercepted SQL is filtered to obtain the query statement, the SQL query statement is parsed, the table name, column name, operation type and condition are extracted, the lexical unit is generated, and the parsed lexical unit is converted into a vector representation as follows:

[0039] v w =embedding(w)

[0040] Among them, v w is the vector representation of word w, constructing the input sequence;

[0041] Build an LSTM model, including the forgetting layer, input layer, LSTM layer, and output layer;

[0042] Extract known dependencies from historical SQL queries and log data as training data, calculate the output, and use the cross entropy loss function to calculate the error loss:

[0043] loss=-∑y i log(P i )

[0044] The gradient is calculated by back-propagation algorithm:

[0045]

[0046] Use the Adam optimizer to update the model parameters and minimize the loss function a:

[0047]

[0048] Among them, y i is the true label of the i-th metadata, P i is the predicted probability output of the i-th metadata, η is the learning rate, and i is the variable index;

[0049] Convert new SQL queries or log data into vector sequences and input them into the trained LSTM model;

[0050] The value of the LSTM model output gate is used as the input value to predict the dependency relationship between data entities and build a data lineage relationship graph;

[0051] According to the data lineage relationship diagram, the constructed data lineage relationship is stored in the Neo4j graph database, including tables, fields, and operation types.

[0052] As a preferred solution of the automatic metadata management method described in the present invention, the predicted dependency relationship includes:

[0053] Dependency relationships are determined through a three-stage dependency model;

[0054] In the first stage of the input model, the similarity S between data is calculated:

[0055]

[0056] Among them, x i and x j represents the i-th and j-th data entities, ε is the scaling factor of similarity, which controls the influence of distance on similarity; ∥d i -d j ∥ indicates data object d i and d j The Euclidean distance of the feature vector, p is the power parameter in the distance calculation, ζ is the weight of the feature contribution, which determines the impact of metadata on the similarity calculation, U is the total number of data entities, and i and j are variable indices;

[0057] After obtaining the similarity S, the value of the LSTM model output gate is used as the input value for the second stage of temporal dynamic modeling D ij (t), the dynamic dependency D ij (t) Integrate to generate the third-stage global dependency matrix E:

[0058]

[0059] Among them, E represents the dependency relationship between all data entities, F is the normalization factor, which ensures that the value of the matrix is ​​in the range of [0,1]; I is the indicator function, which returns 1 when the condition is met, otherwise it is 0; S y is the similarity threshold, which determines the entities that are considered to be related; D ij (t) represents x at time t i and x j Dynamic dependencies between

[0060] When E>0.75, there is a dependency relationship, and when E≤0.75, there is no dependency relationship.

[0061] Another object of the present invention is to provide an automated metadata management system, which realizes the automated collection, management, development and monitoring of metadata to improve the efficiency and quality of data governance and support data-driven business decisions and process optimization. The system solves technical problems such as data source diversity adaptation, metadata collection automation, metadata model management, data development and processing efficiency, data lineage tracking and abnormal data processing. Through these solutions, the system aims to reduce the complexity and cost of metadata management, improve the automation level and efficiency of data governance, support the improvement of data quality and data security, and provide an accurate and reliable data foundation for data analysis and business decision-making.

[0062] To solve the above technical problems, the present invention provides the following technical solutions: an automated metadata management system, comprising: a data collection and adaptation module, responsible for adapting and collecting metadata from different data sources, automatically collecting matching metadata into the metadata database according to the meta-model and collection task configuration by configuring the data source to connect to the database; a meta-model management module, responsible for defining and managing the meta-model library, automatically collecting metadata information, and performing verification to determine whether it needs to be stored or upgraded; a data development and processing module, developing and processing the collected data, including data extraction, conversion, loading into the data warehouse, and performing data enhancement and implicit feature recognition; a blood relationship management module, constructing data blood relationships and predicting dependencies.

[0063] As a preferred solution of the automatic metadata management system described in the present invention, the predicted dependency relationship includes:

[0064] The data collection and adaptation module includes a data source adaptation unit, a collection task management unit and a metadata collection unit. The data source adaptation unit provides connection information to the collection task management unit, and the collection task management unit calls the metadata collection unit to collect data.

[0065] The metamodel management module includes a metamodel library management unit, a metadata verification unit and a metadata version control unit. The metadata collection unit transmits the collected metadata to the metadata verification unit, the metadata verification unit interacts with the metamodel library management unit, and the metadata version control unit performs version management according to the verification result.

[0066] The data development and processing module includes a data extraction unit, a data conversion unit, a data loading unit, a data enhancement unit and a feature recognition unit. After the data extraction unit extracts data, it is transferred to the data conversion unit. The data processed by the data conversion unit is loaded into the data warehouse by the data loading unit. The data enhancement unit and the feature recognition unit provide support in the data processing process.

[0067] The blood relationship management module includes a blood relationship construction unit and a dependency relationship prediction unit. The blood relationship construction unit generates data blood relationship through an analysis tool, and the dependency relationship prediction unit predicts dependency relationship using this information.

[0068] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the method described in the automatic metadata management system configuration are implemented.

[0069] A computer-readable storage medium stores a computer program thereon, wherein when the computer program is executed by a processor, the steps of the method described in the automated metadata management system configuration are implemented.

[0070] Beneficial effects of the present invention: The present invention reduces manual intervention and improves the efficiency of metadata management through an automated metadata collection and management method. By defining a unified meta-model and automatically establishing the relationship between metadata, the consistency and accuracy of metadata are ensured. By automatically parsing SQL scripts and establishing the blood relationship between metadata tables, the flow and dependency of data between different tables can be clearly displayed, which facilitates data management and tracing. The system can adapt to different types of data sources and application scenarios, and has strong adaptability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0072] Figure 1 An overall flow chart of an automatic metadata management method based on model matching provided by one embodiment of the present invention.

[0073] Figure 2 A metadata intelligent management flow chart of an automated metadata management method based on model matching provided by one embodiment of the present invention.

[0074] Figure 3 A flowchart of metadata verification and upgrading of an automated metadata management method based on model matching provided by one embodiment of the present invention.

[0075] Figure 4 A system solution module diagram of an automated metadata management system based on model matching provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0077] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0078] Example 1, reference Figure 1-Figure 3 , which is the first embodiment of the present invention, provides an automatic metadata management method, comprising:

[0079] S1: Adapt the data source and connect to the database to collect metadata that matches the metamodel.

[0080] like Figure 2 ,Furthermore, metadata collection includes data source definition, ,collection task configuration, collection task execution and metadata information storage.

[0081] The collection task configuration includes data source, metamodel classification, collection strategy, automatic storage, collection frequency and collection content. After the collection task is released, a scheduled task for the collection task can be generated. After the specified time is reached, the collection task starts automatically and collects and stores metadata according to the configured metamodel. The collection content is when the add operation is clicked, the configured metamodel is selected from the metamodel library. After the metamodel is added, the collection task will collect the metadata information of the configured metamodel.

[0082] The meta-model is classified as a relational database, and the acquisition strategy includes full update or incremental update.

[0083] The definition of the metamodel library includes the attributes of the metamodel, the combination relationship of the metamodel and the dependency relationship of the metamodel.

[0084] The meta-model includes tables, fields, primary keys, unique constraints, foreign keys, indexes, views, libraries, check restrictions, function parameters, functions, stored procedure parameters, stored procedures, triggers, table spaces, and table partitions.

[0085] The attributes of the metamodel include maintenance according to the characteristics corresponding to the metamodel type, and the attributes of the table include table space, table type, and comments.

[0086] The combination relationship of the meta-models includes constructing sub-meta-models of the current meta-model.

[0087] The dependency relationship of the metamodel includes associated metamodels affected by the current metamodel.

[0088] For example, in the metamodel of views and tables, if the field length, field type and other attributes of a table change, the field length and field type of the view created based on the current table will also change, that is, the dependency relationship of the metamodel of the table includes the view.

[0089] The data source definition mainly includes the database name, type, address, user name and password. The platform uses the DriverManager, Connection and DatabaseMetaData classes provided by Java native JDBC to extract all metadata information of tables and fields in the database.

[0090] The DriverManager class provides a getConnection method for obtaining a database connection. You can obtain a database connection object by passing the database address, user name, and password into the method according to the configuration information. You can use the getMetaData method provided in the Connect class to obtain a databaseMetaData object. The databaseMetaData contains all metadata information about database tables, fields, views, etc.

[0091] S2: Define the metamodel library, collect the current metadata information of the metamodel from the database; verify whether the collected metadata needs to be stored and upgraded.

[0092] like Figure 3 Furthermore, when the collection task is started, the metadata collected before this collection task is queried and the metadata object A is obtained. The id of the metadata object is used as the keyword to store it in the Map collection provided by Java. After all metadata records are stored in the Map, the metadata collection map is obtained:

[0093] map = {ID i : A i |A i ∈A}

[0094] Among them, A is the metadata object set queried before this collection task, map is the metadata object set, and A i is the i-th metadata object, ID i is the id of the i-th metadata object, and i is the variable index.

[0095] The metadata collection task obtains the metadata information B corresponding to the meta-model from the databaseMetaData object according to the meta-model, and generates the metadata object B from the metadata information j , get metadata object B j id as the keyword ID j , query whether the metadata collection map contains the meta ID j :

[0096] When the map does not contain an ID j When metadata B jDoes not exist in the metadata database, perform data insertion operations.

[0097] When the map contains ID j When the metadata B j and the metadata object map (ID j ) are equal, when B j =map(ID j ), no processing is required. j ≠map(ID j ), then the map(ID j ) is recorded as a historical version, and the newly collected metadata B j It is stored in the database as the latest version, and the version number is increased by 1.

[0098]

[0099] Where s1 and s2 are two metadata objects. If the metadata objects are equal, 1 is returned; if they are not equal, 0 is returned.

[0100] val=δ(map(ID j ),B j )

[0101] Where val is map(ID j ) and Bj, if it is 1, they are equal, if it is 0, they are not equal, B j is the jth metadata in the metadata information B collected according to the meta-model, ID j Metadata B j id,map(ID j ) is and B j The metadata record with the same id as , j is the variable index.

[0102] S3: Automatically collect data into data warehouse for data development.

[0103] Furthermore, all data in the metadata corresponding table is automatically extracted based on the data source configuration and metadata information and stored in the data warehouse. The data warehouse is divided into four layers: ODS, DWD, DWS, and ADS.

[0104] The metadata collected in the system is determined according to the data source, and the provided output component automatically writes the collected results to the ODS layer of the specified data warehouse, cleans and converts them, and automatically writes them to the DWD layer through the output component; the data governance module performs preliminary aggregation and calculation based on user data and writes them to the DWS layer of the specified data warehouse, and performs high-dimensional aggregation and calculation on the data in the DWS layer again and writes them to the ADS layer, and automatically generates various charts through the chart components provided by the system and displays them on the web page.

[0105] It should be noted that in the process of data development in the data warehouse, the relationship between metadata tables and fields is established, SpringAOP is used to intercept SQL execution statements, and the intercepted SQL statements are pushed to the Kafka message queue, the data relationship is automatically identified, the intercepted SQL is filtered to obtain query statements, the SQL query statements are parsed, the table name, column name, operation type and condition are extracted, lexical units are generated, and the parsed lexical units are converted into vectors as follows:

[0106] v w =embedding(w)

[0107] Among them, v w It is the vector representation of word w, constructing the input sequence.

[0108] Construct an LSTM model, including a forgetting layer, an input layer, an LSTM layer, and an output layer.

[0109] The forgetting layer:

[0110] g t =Sig(G f ·[h t-1 ,z t ]+b f )

[0111] Among them, g t is the activation value of the forget gate, G f is the weight matrix of the forget gate, h t-1 is the hidden state at the previous moment, z t is the input vector at the current moment, b f Bias vector for the forget gate.

[0112] The input layer:

[0113] in t =Sig(W i ·[h t-1 ,z t ]+b i )

[0114]

[0115] Among them, t is the activation value of the input gate, is the candidate cell state, W i and W C is the weight matrix of the input gate and candidate cell state, b i and b c is the bias vector between the input gate and the candidate cell state.

[0116] Cell status update:

[0117]

[0118] Among them, C t is the cell state at the current moment, C t-1 is the cell state at the previous moment.

[0119] The output layer:

[0120] o t =Sig(W o ·[h t-1 ,z t ]+b o )

[0121] h t =o t *tanh(C t )

[0122] Among them, t is the activation value of the output gate, h t is the hidden state at the current moment, W o is the weight matrix of the output gate, b o is the bias vector for the output gate.

[0123] Determine the metadata collected in the system according to the data source, automatically associate it with the table data by selecting the collected table name and field name in the metadata, and specify the collected metadata data set map = {ID i}, enhance the metadata information:

[0124]

[0125] Among them, map' is the enhanced metadata set, A j For any metadata object, α j is the scaling parameter of the jth metadata, β j is the translation parameter of the jth metadata, μ j is the mean value of the jth metadata, σ j is the standard deviation of the j-th metadata.

[0126] The dynamic data classification algorithm is used to identify the implicit features of the data and to use the enhanced map' ij Perform sparse coding map of implicit features, use sparse representation to train the logistic regression model, and obtain the predicted output P for each sample i :

[0127]

[0128] Among them, y i is the true label of the i-th metadata, P i is the predicted probability output of the i-th metadata, Sig is the Sigmoid function, which is usually used for logistic regression, and the output value is between (0,1); W Z T represents the transpose of the dictionary matrix W to prevent overfitting of the model; n is the total number of metadata sets, i and j are variable indexes, and W Z is the dictionary matrix containing all basis vectors, λ Z is a regularization parameter used to control the sparsity of Z.

[0129] Based on the model's predicted output, filter conditions are configured, and the output component provided by the system automatically writes the collected results to the ODS layer of the specified data warehouse.

[0130] The component provides a data input component. The component configures the filtering conditions of regular expressions, specified values, and custom functions. According to the configured filtering conditions, the data to be cleaned is obtained for cleaning and conversion. The converted and cleaned data is mapped clean Automatically written to the DWD layer by the output component.

[0131] The data governance module performs preliminary aggregation and calculation based on user data, and writes the obtained preliminary aggregation and calculation results to the DWS layer of the specified data warehouse through the output component; according to the user's decision-making needs, the data in the DWS layer is aggregated and calculated again at a high level, and finally written to the ADS layer. Various charts are automatically generated through the chart components provided by the system and displayed on the web page.

[0132] It should be noted that in the process of data development in the data warehouse, the relationship between metadata tables and fields is established, SpringAOP is used to intercept SQL execution statements, and the intercepted SQL statements are pushed to the Kafka message queue, the data relationship is automatically identified, the intercepted SQL is filtered to obtain query statements, the SQL query statements are parsed, the table name, column name, operation type and condition are extracted, lexical units are generated, and the parsed lexical units are converted into vectors as follows:

[0133] v w =embedding(w)

[0134] Among them, v w is the vector representation of word w, constructing the input sequence;

[0135] Build an LSTM model, including the forgetting layer, input layer, LSTM layer, and output layer;

[0136] Extract known dependencies from historical SQL queries and log data as training data, calculate the output and use the cross entropy loss function to calculate the error loss:

[0137] loss=-∑y i log(P i )

[0138] The gradient is calculated by back-propagation algorithm:

[0139]

[0140] Use the Adam optimizer to update the model parameters and minimize the loss function a:

[0141]

[0142] Among them, P i is the predicted probability output of the i-th metadata, η is the learning rate; the new SQL query or log data is converted into a vector sequence and input into the trained LSTM model, the value of the LSTM model output gate is used as the input value to predict the dependency relationship between data entities, and a data lineage relationship graph is constructed. The constructed data lineage relationship is stored in the Neo4j graph database, including tables, fields, and operation types.

[0143] S4: When a problem occurs in data development, the indicator information specified by the user is obtained and the blood relationship is automatically constructed to locate abnormal data and process it.

[0144] Furthermore, the dependency relationship is determined through a three-stage dependency model. In the first stage of the input model, the similarity S between the data is calculated:

[0145]

[0146] Among them, x i and x j Represents the i-th and j-th metadata entities, ε is the scaling factor of similarity, which controls the influence of distance on similarity. ∥d i -d j ∥ indicates data object d i and d jThe Euclidean distance of the feature vector, p is the power parameter in the distance calculation, and ζ is the weight of the feature contribution, which determines the impact of metadata on the similarity calculation.

[0147] After obtaining the similarity S, the value of the LSTM model output gate is used as the input value for the second stage of temporal dynamic modeling:

[0148]

[0149] Among them, D ij (t) represents x at time t i and x j The dynamic dependency between them, ρ is the time decay coefficient, t is the current time, t tep It is the reference time point, usually the start time of the analysis.

[0150] The dynamic dependency D ij (t) Perform the third stage integration to generate the global dependency matrix E:

[0151]

[0152] Among them, E represents the dependency relationship between all data entities, F is the normalization factor, which ensures that the value of the matrix is ​​in the range of [0,1]. I is the indicator function, which returns 1 when the condition is met, otherwise it is 0. y is the similarity threshold that determines the entities that are considered related.

[0153] When E>0.75, there is a dependency relationship, and when E≤0.75, there is no dependency relationship.

[0154] Constructing data lineage relationship graph E t :

[0155] E t =(x,bi),bi={(x i ,x j )|E ij >0}

[0156] Among them, E t represents the connection between entities, x represents the data entity in the graph, bi represents the dependency relationship between data entities, E ij is an element in the global dependency matrix, representing x i and x j of dependence.

[0157] Example 2, reference Figure 4, is an embodiment of the present invention, which provides an automated metadata management system, including: a data collection and adaptation module 100, a metamodel management module 200, a data development and processing module 300, and a blood relationship management module 400.

[0158] The data collection and adaptation module 100 is used to adapt and collect metadata from different data sources, and automatically collect the matching metadata into the metadata database by configuring the data source to connect to the database according to the meta-model and collection task configuration.

[0159] Specifically, it includes a data source adaptation unit, a collection task management unit, and a metadata collection unit;

[0160] The data source adapter unit provides connection information to the collection task management unit, and the collection task management unit calls the metadata collection unit to collect data.

[0161] The metamodel management module 200 is used to define and manage the metamodel library, automatically collect metadata information, and perform verification to determine whether it needs to be stored or upgraded.

[0162] Specifically, it includes a meta-model library management unit, a metadata verification unit, and a metadata version control unit;

[0163] The metadata collection unit transfers the collected metadata to the metadata verification unit, the metadata verification unit interacts with the meta-model library management unit, and the metadata version control unit performs version management according to the verification result.

[0164] The data development and processing module 300 is used to develop and process the collected data, including data extraction, conversion, loading into the data warehouse, and data enhancement and implicit feature recognition.

[0165] Specifically, a data extraction unit, a data conversion unit, a data loading unit, a data enhancement unit, and a feature recognition unit;

[0166] After the data extraction unit extracts the data, it is passed to the data conversion unit. The data processed by the data conversion unit is loaded into the data warehouse by the data loading unit. The data enhancement unit and the feature recognition unit provide support in the data processing process.

[0167] The blood relationship management module 400 is used to build data blood relationships and predict dependency relationships.

[0168] Specifically, it includes a blood relationship construction unit and a dependency relationship prediction unit;

[0169] The blood relationship construction unit generates data blood relationships through analysis tools, and the dependency prediction unit uses this information to predict dependencies.

[0170] Embodiment 3, an embodiment of the present invention, is different from the first two embodiments in that:

[0171] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0172] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0173] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0174] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

Claims

1. An automated metadata management method, characterized in that: include, Adapt the data source and connect to the database, and collect metadata that matches the metamodel; Define the metamodel library and collect the current metadata information of the metamodel in the database; Verify whether the collected metadata needs to be stored and updated; Automatically collect data from data warehouse for data development; Obtain user-specified indicator information to build kinship relationships and predict dependency relationships; The said establishing of blood relationship comprises: In the process of data development in the data warehouse, establish the blood relationship between metadata tables and fields, use SpringAOP to intercept SQL execution statements, and push the intercepted SQL statements to the Kafka message queue to automatically identify data blood relationships; The intercepted SQL is filtered to obtain the query statement, the SQL query statement is parsed, the table name, column name, operation type and condition are extracted, the lexical unit is generated, and the parsed lexical unit is converted into a vector representation as follows: v w =embedding(w) Among them, v w is the vector representation of word w, constructing the input sequence; Build an LSTM model, including the forgetting layer, input layer, LSTM layer, and output layer; Extract known dependencies from historical SQL queries and log data as training data, calculate the output, and use the cross entropy loss function to calculate the error loss: loss=-∑y i log(P i ) The gradient is calculated by back-propagation algorithm: Use the Adam optimizer to update the model parameters and minimize the loss function a: Among them, y i is the true label of the i-th metadata, P i is the predicted probability output of the i-th metadata, η is the learning rate, and i is the variable index; Convert new SQL queries or log data into vector sequences and input them into the trained LSTM model; The value of the LSTM model output gate is used as the input value to predict the dependency relationship between data entities and build a data lineage relationship graph; According to the data lineage relationship diagram, the constructed data lineage relationship is stored in the Neo4j graph database, including tables, fields, and operation types; The predicted dependencies include: Dependency relationships are determined through a three-stage dependency model; In the first stage of the input model, the similarity S between data is calculated: Among them, x i and x j represents the i-th and j-th data entities, ε is the scaling factor of similarity, which controls the influence of distance on similarity; ∥d i -d j ∥ indicates data object d i and d j The Euclidean distance of the feature vector, p is the power parameter in the distance calculation, ζ is the weight of the feature contribution, which determines the impact of metadata on the similarity calculation, U is the total number of data entities, and i and j are variable indices; After obtaining the similarity S, the value of the LSTM model output gate is used as the input value for the second stage of temporal dynamic modeling D ij (t), the dynamic dependency D ij (t) Integrate to generate the third-stage global dependency matrix E: Among them, E represents the dependency relationship between all data entities, F is the normalization factor, which ensures that the value of the matrix is ​​in the range of [0,1]; I is the indicator function, which returns 1 when the condition is met, otherwise it is 0; S y is the similarity threshold, which determines the entities that are considered to be related; D ij (t) represents x at time t i and x j Dynamic dependencies between When E>0.75, there is a dependency relationship, and when E≤0.75, there is no dependency relationship.

2. The method of automated metadata management according to claim 1, characterized in that: The metadata matching the meta-model is collected, including: Metadata collection includes data source definition, collection task configuration, collection task execution and metadata information storage; The collection task configuration includes data source, metamodel classification, collection strategy, automatic storage, collection frequency and collection content; After the collection task is released, a scheduled task is generated. When the specified time comes, the collection task starts automatically and collects and stores metadata according to the configured meta-model. The collection content is that when clicking the add operation, the configured meta-model is selected from the meta-model library. After the meta-model is added, the collection task will collect the metadata information of the configured meta-model.

3. The method of automated metadata management according to claim 2, characterized in that: The verification of whether the collected metadata needs to be stored and updated includes: When the collection task is started automatically, query the metadata collected before this collection task and obtain the metadata object; Store the IDs of all metadata objects as keywords in the metadata collection map; The metadata collection task obtains metadata information corresponding to the metamodel from the databaseMetaData object according to the metamodel; The metadata information generates a metadata object, defines the id of the metadata object as the keyword, and queries whether the metadata collection map contains the keyword of the current metadata object; When the map does not contain a keyword, the current metadata object does not exist in the metadata database, and data is inserted.

4. The method of automated metadata management according to claim 3, characterized in that: The verification of whether the collected metadata needs to be stored and updated also includes: When the map contains keywords, it determines whether the current metadata object is equal to the metadata object in the metadata collection map; If the current metadata object is equal to the metadata object in the metadata collection map, no processing is required; If the current metadata object is not equal to the metadata object in the metadata collection map, the metadata object in the metadata database is recorded as a historical version, and the current metadata object is stored in the database as the latest version, and the version number is increased by 1.

5. The method of automated metadata management according to claim 4, characterized in that: The data development includes, Data collection is performed through tables and fields in the associated metadata database; Store all data in the metadata corresponding table into the data warehouse; The data warehouse includes ODS, DWD, DWS, and ADS; Determine the metadata collected in the system based on the data source; The provided output component automatically writes the collected results to the ODS layer of the specified data warehouse; Cleans and converts the data and automatically writes it to the DWD layer through the output component; Perform preliminary aggregation and calculation based on user data and write it to the DWS layer of the specified data warehouse; The data in the DWS layer is aggregated and calculated again at a high level and written into the ADS layer; Automatically generate charts through chart components and display them on web pages.

6. An automated metadata management system, characterized in that: include, The data collection and adaptation module (100) is used to adapt and collect metadata from different data sources, and automatically collect the matching metadata into the metadata database by configuring the data source to connect to the database according to the meta-model and collection task configuration; The metamodel management module (200) is used to define and manage the metamodel library, automatically collect metadata information, and perform verification to determine whether it needs to be stored or upgraded; A data development and processing module (300) is used to develop and process the collected data, including data extraction, conversion, loading into a data warehouse, and data enhancement and implicit feature recognition; The blood relationship management module (400) is used to build data blood relationship and predict dependency relationship, specifically: In the process of data development in the data warehouse, establish the blood relationship between metadata tables and fields, use SpringAOP to intercept SQL execution statements, and push the intercepted SQL statements to the Kafka message queue to automatically identify data blood relationships; The intercepted SQL is filtered to obtain the query statement, the SQL query statement is parsed, the table name, column name, operation type and condition are extracted, the lexical unit is generated, and the parsed lexical unit is converted into a vector representation as follows: v w =embedding(w) Among them, v w is the vector representation of word w, constructing the input sequence; Build an LSTM model, including the forgetting layer, input layer, LSTM layer, and output layer; Extract known dependencies from historical SQL queries and log data as training data, calculate the output, and use the cross entropy loss function to calculate the error loss: loss=-∑y i log(P i ) The gradient is calculated by back-propagation algorithm: Use the Adam optimizer to update the model parameters and minimize the loss function a: Among them, y i is the true label of the i-th metadata, P i is the predicted probability output of the i-th metadata, η is the learning rate, and i is the variable index; Convert new SQL queries or log data into vector sequences and input them into the trained LSTM model; The value of the LSTM model output gate is used as the input value to predict the dependency relationship between data entities and build a data lineage relationship graph; According to the data lineage relationship diagram, the constructed data lineage relationship is stored in the Neo4j graph database, including tables, fields, and operation types; The predicted dependencies include: Dependency relationships are determined through a three-stage dependency model; In the first stage of the input model, the similarity S between data is calculated: Among them, x i and x j represents the i-th and j-th data entities, ε is the scaling factor of similarity, which controls the influence of distance on similarity; ∥d i -d j ∥ indicates data object d i and d j The Euclidean distance of the feature vector, p is the power parameter in the distance calculation, ζ is the weight of the feature contribution, which determines the impact of metadata on the similarity calculation, U is the total number of data entities, and i and j are variable indices; After obtaining the similarity S, the value of the LSTM model output gate is used as the input value for the second stage of temporal dynamic modeling D ij (t), the dynamic dependency D ij (t) Integrate to generate the third-stage global dependency matrix E: Among them, E represents the dependency relationship between all data entities, F is the normalization factor, which ensures that the value of the matrix is ​​in the range of [0,1]; I is the indicator function, which returns 1 when the condition is met, otherwise it is 0; S y is the similarity threshold, which determines the entities that are considered to be related; D ij (t) represents x at time t i and x j Dynamic dependencies between When E>0.75, there is a dependency relationship, and when E≤0.75, there is no dependency relationship.

7. An automated metadata management system as claimed in claim 6, characterized in that: The data acquisition and adaptation module, It includes a data source adapter unit, a collection task management unit and a metadata collection unit. The data source adapter unit provides connection information to the collection task management unit, and the collection task management unit calls the metadata collection unit to collect data. The metamodel management module includes a metamodel library management unit, a metadata verification unit and a metadata version control unit. The metadata collection unit transmits the collected metadata to the metadata verification unit. The metadata verification unit interacts with the metamodel library management unit. The metadata version control unit performs version management according to the verification result. The data development and processing module includes a data extraction unit, a data conversion unit, a data loading unit, a data enhancement unit and a feature recognition unit. After the data extraction unit extracts data, it is transferred to the data conversion unit. The data processed by the data conversion unit is loaded into the data warehouse by the data loading unit. The data enhancement unit and the feature recognition unit provide support in the data processing process. The blood relationship management module includes a blood relationship construction unit and a dependency relationship prediction unit. The blood relationship construction unit generates data blood relationship through an analysis tool, and the dependency relationship prediction unit predicts dependency relationship using this information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the automated metadata management method described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the automated metadata management method described in any one of claims 1 to 5 are implemented.

Images