Metadata synchronization methods, devices, storage media, electronic devices and program products

Abstract

This application discloses a metadata synchronization method, apparatus, storage medium, electronic device, and program product, relating to the fields of big data and artificial intelligence. The method includes: acquiring a metadata definition language script (MDL), which describes changes in the table structure of a database; preprocessing the MDL to obtain a target MDL; parsing the target MDL to obtain at least one type of metadata information, the parsing operation including: determining the business meaning of each field in the target MDL through named entity recognition rules, and performing syntax parsing on the target MDL based on a neural network model; and synchronizing each type of metadata information to the target system according to its type. This application solves the technical problem of low metadata management efficiency in the prior art.

Description

Technical Field

[0001] This application relates to the fields of big data and artificial intelligence, and more specifically, to a metadata synchronization method, apparatus, storage medium, electronic device, and program product. Background Technology

[0002] As the value of data elements continues to increase, users are placing higher demands on metadata management. Currently, metadata management typically involves multiple stages, including data quality verification, data standard verification, and data asset information generation.

[0003] However, metadata management systems in related technologies often disperse these functions across different modules or systems. This results in users needing to perform separate data quality verification, data standard verification, data entry into the data lake, and data asset information generation operations after entering metadata. This process is cumbersome and inefficient, easily leading to inconsistencies in metadata information, potentially impacting downstream data and creating hidden production risks. Because the business needs for metadata management functions are supplied by different departments, management and entry are often handled in different systems. Users need to perform multiple operations to complete metadata verification and asset information generation, increasing time and manpower costs. In particular, developers not only need to create metadata in the code but also enter it into the metadata management system. The lack of automation mechanisms prevents all related operations from being completed at once, impacting overall data governance efficiency. Furthermore, the approval processes for adding new metadata, scattered across different business systems, are also inconsistent.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a metadata synchronization method, apparatus, storage medium, electronic device, and program product to at least solve the technical problem of low metadata management efficiency in the prior art.

[0006] According to one aspect of the embodiments of this application, a metadata synchronization method is provided, comprising: obtaining a metadata definition language script, wherein the metadata definition language script is used to describe changes in the table structure of a database; preprocessing the metadata definition language script to obtain a target metadata definition language script; performing a parsing operation on the target metadata definition language script to obtain at least one metadata information, wherein the parsing operation includes: determining the business meaning of each field in the target metadata definition language script through named entity recognition rules, and performing syntax parsing on the target metadata definition language script based on a neural network model; and synchronizing each type of metadata information to a target system according to the type of each metadata information.

[0007] Optionally, preprocessing includes at least one of the following operations: text normalization, used to convert all characters in the metadata definition language script into a uniform form; comment deletion, used to detect and delete comment information in the metadata definition language script; character processing, used to merge newline characters in the metadata definition language script and remove redundant spaces; syntax formatting, used to uniformly sort and format field names, data types, and constraints in the metadata definition language script; and compatibility adjustment, used to convert syntax elements in the metadata definition language script into a predefined general format according to the version and syntax differences of different database management systems.

[0008] Optionally, before parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: detecting the type of each statement in the target metadata definition language script, wherein the statement type is a first type statement, a second type statement, or a third type statement, the first type statement is used to create a table in the database, the second type statement is used to modify a table in the database, and the third type statement is used to delete a table in the database.

[0009] Optionally, the target metadata definition language script is parsed to obtain at least one metadata information, including: identifying the names of each table in the database using regular expressions based on the type of each statement in the target metadata definition language script; using a recursive descent parsing algorithm to parse the fields related to the database tables in the target metadata definition language script one by one according to the named entity recognition rules to obtain the business meaning of each field; and determining the metadata information based on the business meaning of each field.

[0010] Optionally, the target metadata definition language script is parsed to obtain at least one metadata information, including: constructing an abstract syntax tree about the target metadata definition language script using a syntax analysis tool, wherein the abstract syntax tree represents the hierarchical relationship between the table structure, fields, and constraints of the target metadata definition language script through a tree structure; and determining the metadata information based on the abstract syntax tree.

[0011] Optionally, the target metadata definition language script is parsed to obtain at least one metadata information, including: obtaining the primary key, non-null value, default value and index information in the target metadata definition language script; generating parsing constraints based on the primary key, non-null value, default value and index information; and parsing the target metadata definition language script according to the parsing constraints to obtain at least one metadata information.

[0012] Optionally, the metadata synchronization method further includes: parsing the column operation instructions corresponding to the second type of statement when the target metadata definition language script includes a second type of statement, wherein the column operation instructions are used to represent instructions to add or modify columns in a table in the database.

[0013] Optionally, after parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: sending the obtained metadata information to the user interface; and determining the verification result of the metadata information based on the user instruction information returned by the user interface.

[0014] Optionally, after parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: when any metadata information is detected to meet at least one of the following conditions, sending the metadata information to the auditing program for verification and auditing; the first condition is used to indicate that the metadata information involves adjustments to table structure fields; the second condition is used to indicate that the metadata information involves changes to field names or changes to data asset information.

[0015] According to another aspect of the embodiments of this application, a metadata synchronization device is also provided, comprising: a script acquisition unit for acquiring a metadata definition language script, wherein the metadata definition language script is used to describe changes in the table structure of a database; a preprocessing unit for preprocessing the metadata definition language script to obtain a target metadata definition language script; a parsing operation unit for parsing the target metadata definition language script to obtain at least one metadata information, wherein the parsing operation includes: determining the business meaning of each field in the target metadata definition language script through named entity recognition rules, and performing syntax parsing on the target metadata definition language script based on a neural network model; and an information synchronization unit for synchronizing each type of metadata information to a target system according to the type of each type of metadata information.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described metadata synchronization method.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors perform the above-described metadata synchronization method.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that implement the above-described metadata synchronization method when executed by a processor.

[0019] In this embodiment, the metadata management system first obtains a metadata definition language script (MDL), which describes changes in the database table structure. The MDL is then preprocessed to obtain a target MDL. Next, the target MDL is parsed to obtain at least one type of metadata information. The parsing process includes: determining the business meaning of each field in the target MDL using named entity recognition rules; performing syntax parsing on the target MDL based on a neural network model; and finally, synchronizing each type of metadata information to the target system according to its type.

[0020] As shown above, preprocessing not only cleans and standardizes the script content but also improves the accuracy and efficiency of subsequent parsing. For example, through text standardization and comment removal, metadata management systems can quickly remove redundant information from metadata definition language scripts and simplify the script structure. Syntax formatting and compatibility adjustments help ensure the consistency and readability of scripts across different database management systems. Preprocessing also helps reduce parsing errors and processing delays caused by inconsistent metadata definition language script formats or complex structures.

[0021] After preprocessing, the metadata management system can parse the target metadata definition language script using named entity recognition rules and neural network model-based syntax parsing technology. Named entity recognition rules automatically identify the business meaning of each field in the target metadata definition language script, while the neural network model handles complex syntactic structures, ensuring accuracy and completeness in parsing. Combining named entity recognition rules and neural network models in this AI-driven parsing process not only improves parsing efficiency but also enables the handling of complex metadata definition language script statement structures. After parsing, the metadata management system synchronizes each type of metadata information to the target system, reducing manual intervention and ensuring consistency and timeliness of metadata information across different systems. This improves the overall efficiency of metadata management and solves the technical problem of low metadata management efficiency in existing technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a schematic diagram of an optional metadata synchronization method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram illustrating the principle of an optional metadata synchronization method according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a statement parsing module in an optional metadata synchronization method according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an optional metadata synchronization device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, a method embodiment for metadata synchronization is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] According to the embodiments of this application, a metadata management system can be used as the execution subject of the metadata synchronization method of this application embodiment. The system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the method execution subject.

[0031] Figure 1 This is a metadata synchronization method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0032] Step S101: Obtain the metadata definition language script, wherein the metadata definition language script is used to describe changes in the table structure of the database.

[0033] Optionally, the metadata management system may first need to obtain a specific script file written in a metadata definition language (MDL). MDL is a language used to define and modify database structures, describing operations such as table creation, modification, and deletion. For example, when developers need to add a new table or adjust fields in an existing table, they can write a MDL script. The MDL script records detailed changes to the table structure, including table names, field names, data types, constraints, and other information. By obtaining the MDL script, the metadata management system can understand the specific changes to the database structure, enabling subsequent parsing and synchronization operations, thus ensuring the consistency and accuracy of the metadata.

[0034] Step S102: Preprocess the metadata definition language script to obtain the target metadata definition language script.

[0035] Optionally, the obtained raw metadata definition language script undergoes a series of preprocessing operations, such as formatting and cleaning, to make it more suitable for subsequent parsing and processing. Preprocessing operations may include text standardization (e.g., unifying character case), deleting comments, merging newlines, removing redundant spaces, standardizing field names, data types, and constraint sorting and formatting, and making compatibility adjustments based on different database management system versions and syntax differences. This simplifies the structure of the metadata definition language script, reduces formatting issues that may affect parsing accuracy, and ensures that the metadata definition language script can be efficiently and accurately processed by subsequent parsing modules.

[0036] Step S103: Perform a parsing operation on the target metadata definition language script to obtain at least one metadata information. The parsing operation includes: determining the business meaning of each field in the target metadata definition language script through named entity recognition rules, and performing syntax parsing on the target metadata definition language script based on a neural network model.

[0037] Optionally, named entity recognition refers to a natural language processing technique used to identify entities with specific meanings from text. In metadata definition language scripts, these entities can be field names, data types, constraints, etc. Through predefined named entity recognition rules, the metadata management system can automatically annotate and understand the business meaning of these fields. For example, the field name "customer_id" can be recognized as "customer identifier," and the field type "varchar(20)" can be recognized as a string type with a length of 20. The recognition process improves the accuracy of parsing.

[0038] Optionally, after identifying the business meaning of the fields, the metadata management system further utilizes syntax parsing technology based on neural network models to process the target metadata definition language script. Neural network models, especially pre-trained models, are capable of deep parsing of complex language structures. Neural network models can process syntactic elements in the target metadata definition language script, such as table names, field definitions, and constraints, and construct abstract syntax trees to clearly represent the hierarchical relationships between these elements. For example, for a table definition statement containing multiple fields and constraints, the neural network model can accurately parse the detailed information of each field and organize this information into a structured form, handling complex metadata definition language script statement structures and ensuring the integrity and accuracy of metadata information. This application embodiment can combine a bidirectional transformer encoder representation model with a conditional random field model and syntax analysis technology to achieve millisecond-level parsing of the metadata definition language script, thereby improving efficiency. It supports concurrent parsing of multiple DDL statements, suitable for metadata synchronization scenarios in large-scale data warehouses or data lakes. It generates data quality rules, data standard verification, and data asset information in one go, avoiding repeated data entry by users in multiple systems and saving operation time.

[0039] Step S104: Synchronize each type of metadata information to the target system according to the type of each metadata information.

[0040] Optionally, by using metadata definition language scripts such as metadata code submitted by developers, the system detects the metadata involved in the metadata code, automatically generates corresponding data quality verification rules, data standard verification rules, and data asset information in one go, and synchronizes them to various metadata modules. This synchronizes the metadata information approval process, simplifying operations and improving the efficiency and consistency of data governance. Data quality verification rules can be synchronized to the data quality management module, data standard verification rules to the data standard management module, and data asset information to the data asset management system. This categorized synchronization process ensures that different types of metadata are correctly applied to their respective systems, thereby improving the efficiency and consistency of data governance. The metadata management system can automatically transmit the parsed metadata information to various relevant systems, reducing the tediousness and error potential of manual operations, while also ensuring the consistency and timeliness of metadata across different systems.

[0041] In one optional embodiment, preprocessing includes at least one of the following operations: text normalization, used to convert all characters in the metadata definition language script into a uniform form; comment deletion, used to detect and delete comment information in the metadata definition language script; character processing, used to merge newline characters in the metadata definition language script and remove redundant spaces in the metadata definition language script; syntax formatting, used to uniformly sort and format field names, data types, and constraints in the metadata definition language script; and compatibility adjustment, used to convert syntax elements in the metadata definition language script into a predefined general format according to the versions and syntax differences of different database management systems.

[0042] Optionally, text normalization operations are used to convert all characters in the Metadata Definition Language (MDL) script into a uniform form. For example, all characters can be standardized to lowercase or uppercase to eliminate parsing errors caused by inconsistent case. Comment removal operations are used to detect and remove comment information in the MDL script. While comments are helpful for users to read the code, they can interfere with the recognition of the script structure during parsing. Removing comments simplifies the script content, making it easier to parse. Character processing operations are used to merge newline characters and remove redundant spaces in the script, helping to clean up unnecessary characters, further simplifying the script structure, and improving the accuracy and efficiency of parsing.

[0043] Optionally, syntax formatting operations are used to uniformly sort and format field names, data types, constraints, etc., in the metadata definition language script, ensuring a more standardized and consistent structure. For example, arranging field names alphabetically and writing data types and constraints in a standard format can improve parsing efficiency and accuracy. Compatibility adjustment operations are used to convert syntax elements in the metadata definition language script into a predefined universal format based on the versions and syntax differences of different database management systems. Since different database management systems may have different syntax requirements, adjustment operations ensure that the metadata definition language script can be correctly understood and processed in different systems, thereby improving the compatibility and universality of the metadata management system.

[0044] In an optional embodiment, before parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: the metadata management system can detect the type of each statement in the target metadata definition language script, wherein the statement type is a first type statement, a second type statement, or a third type statement, the first type statement is used to create a table in the database, the second type statement is used to modify a table in the database, and the third type statement is used to delete a table in the database.

[0045] Optionally, the statement types in the target metadata definition language script are divided into three types: the first type is the CREATE TABLE statement, used to create a new table in the database; the second type is the ALTER TABLE statement, used to modify an existing table in the database, including operations such as adding, modifying, or deleting fields; and the third type is the DROP TABLE statement, used to delete a table in the database. By detecting these statement types, the metadata management system can accurately understand the specific operational intent of the target metadata definition language script.

[0046] Optionally, by detecting the various statement types in the target metadata definition language script, the metadata management system can accurately identify and process changes in the database structure. For example, when the metadata management system detects a CREATE TABLE statement, it knows that a new table needs to be created and can further parse the table's structure and field information. For an ALTER TABLE statement, the metadata management system can identify modifications to existing tables, such as adding, modifying, or deleting fields, and update the metadata information accordingly. Detecting DROP TABLE statements allows the metadata management system to promptly handle table deletion operations, ensuring the consistency and accuracy of metadata. Precise statement type detection not only improves the automation capabilities of the metadata management system but also reduces the need for manual intervention, thereby improving the efficiency and reliability of metadata management.

[0047] In one optional embodiment, the target metadata definition language script is parsed to obtain at least one metadata information, including: the metadata management system can identify the names of each table in the database using regular expressions based on the type of each statement in the target metadata definition language script; then, using a recursive descent parsing algorithm, the fields related to the database tables in the target metadata definition language script are parsed one by one according to the named entity recognition rules to obtain the business meaning of each field; finally, the metadata information is determined based on the business meaning of each field.

[0048] Optionally, when processing the target metadata definition language script, the metadata management system first identifies the names of each table in the database based on the type of each statement in the script. This process is mainly achieved through regular expressions, a text matching tool that can quickly and accurately extract key information such as table names from the script. For example, for the CREATE TABLE statement, a regular expression can match the table name following the CREATE TABLE keyword.

[0049] Optionally, after identifying the table name, the metadata management system can further utilize a recursive descent parsing algorithm, combined with named entity recognition rules, to parse each field in the target metadata definition language script that is related to the database table. The recursive descent parsing algorithm is a top-down parsing method that can effectively handle nested and complex syntax structures. Through named entity recognition rules, the metadata management system can identify the business meaning of each field. For example, the field name `customer_id` can be identified as a customer identifier. The identification process may include not only the field name but also other relevant information such as data type and constraints. Based on the business meaning of each field, the metadata management system determines complete metadata information, facilitating the accuracy and consistency of the metadata.

[0050] In one optional embodiment, the target metadata definition language script is parsed to obtain at least one metadata information, including: the metadata management system can use a syntax analysis tool to construct an abstract syntax tree about the target metadata definition language script, wherein the abstract syntax tree represents the hierarchical relationship between the table structure, fields and constraints of the target metadata definition language script through a tree structure; and then the metadata information is determined based on the abstract syntax tree.

[0051] Optionally, when processing the target metadata definition language script, the metadata management system utilizes syntax analysis tools to construct an Abstract Syntax Tree (AST). An AST is a tree-like structure used to represent the syntactic elements in the target metadata definition language script and the hierarchical relationships between them. Specifically, an AST clearly displays the relationships between table structures, fields, and constraints. For example, the root node of the tree can represent the entire table definition, and its child nodes represent table names, field definitions, and constraints, respectively. Through this tree structure, the system can more intuitively understand and parse the script's syntactic structure.

[0052] Optionally, after constructing the abstract syntax tree (AST), the metadata management system uses the AST to determine metadata information. The AST provides a clear view of the syntax structure, enabling the metadata management system to accurately extract and organize metadata information. For example, the metadata management system can extract information such as table names, field names, data types, and constraints by traversing the AST and store this information in a structured manner. This not only improves the accuracy and efficiency of parsing but also allows for the handling of complex syntax structures, facilitating the integrity and consistency of metadata information.

[0053] In one optional embodiment, the target metadata definition language script is parsed to obtain at least one metadata information, including: the metadata management system can obtain the primary key, non-null value, default value and index information in the target metadata definition language script; then generate parsing constraints based on the primary key, non-null value, default value and index information; finally, the target metadata definition language script is parsed according to the parsing constraints to obtain at least one metadata information.

[0054] Optionally, when processing the target metadata definition language script, the metadata management system first extracts key information from the script, including primary keys, non-null values, default values, and index information. This information can be used to define table behavior and data integrity rules. For example, a primary key uniquely identifies each row in the table, non-null constraints ensure that certain fields must have values, default values ​​provide a default value for a field, and index information improves query efficiency. By obtaining this information from the target metadata definition language script, the metadata management system can gain a more comprehensive understanding of the table's structure and constraints.

[0055] Optionally, after obtaining this key information from the target metadata definition language script, the metadata management system generates parsing constraints based on the primary key, NOT NULL values, default values, and index information. These constraints help the metadata management system understand the specific requirements of the table structure. For example, if a field is defined as a primary key, the metadata management system can generate a constraint to ensure that the value of that field is unique within the table. Similarly, if a field has a NOT NULL constraint, the metadata management system can generate a corresponding constraint to ensure that the field cannot be null when data is inserted. The metadata management system parses the target metadata definition language script based on these parsing constraints, extracting at least one type of metadata information. This metadata information can be used for subsequent synchronization and management operations, facilitating the accuracy and consistency of the metadata.

[0056] For example, the named entity recognition rules can be found in Table 1:

[0057] Table 1

[0058]

[0059] Constraint resolution can be performed according to the following example:

[0060] Primary key: marked as PK;

[0061] Not null: marked as NOT NULL;

[0062] Default value: current_timestamp: Stores the default expression;

[0063] Index: index idx_name (cust_name): Records the index name and fields.

[0064] In an optional embodiment, the metadata synchronization method further includes: if the target metadata definition language script includes a second type of statement, the metadata management system can parse the column operation instructions corresponding to the second type of statement, wherein the column operation instructions are used to represent instructions to add or modify columns in a table in the database.

[0065] Optionally, when processing the target metadata definition language script, the metadata management system pays special attention to the second type of statements contained in the target metadata definition language script, namely the ALTER TABLE statement. This type of statement is used to modify existing tables in the database, including adjusting the table structure. When the metadata management system detects that the target metadata definition language script contains an ALTER TABLE statement, it can further parse the corresponding column operation instructions in the statement. These column operation instructions specifically represent modification operations on the database table, such as adding a new column or modifying the definition of an existing column.

[0066] Optionally, if the column operation instruction is `ADD COLUMN`, the metadata management system can parse out the name, data type, constraints, and other information of the new column and add it to the table structure. If the column operation instruction is `MODIFY COLUMN` or `CHANGE COLUMN`, the metadata management system can parse out the name of the column to be modified, the new data type, the new constraints, and other information, and update the table structure accordingly. The metadata management system can accurately reflect changes in the table structure, facilitating the assurance of the accuracy and consistency of metadata information.

[0067] In an optional embodiment, after parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: the metadata management system can send the obtained metadata information to a user interface. Then, the verification result of the metadata information is determined based on the user instruction information returned by the user interface.

[0068] Optionally, after parsing the target metadata definition language script and extracting the metadata information, the metadata management system can send the metadata information to the user interface. The user interface provides users with an intuitive platform to view and verify the accuracy and completeness of the metadata information. Users can check metadata information, such as table structure, field definitions, and constraints, through the user interface and make necessary modifications or confirmations as needed. User operations on the interface generate user instruction information, which can be received and processed by the metadata management system. The metadata management system determines the verification result of the metadata information based on the user's returned instruction information. For example, if the user confirms that the information is correct, the metadata management system can mark the metadata information as verified; if the user finds errors and makes modifications, the metadata management system can re-verify based on the modified information, thus ensuring the accuracy and reliability of the metadata information and increasing user participation and control over metadata management.

[0069] See Figure 2A metadata management system may include a statement parsing module, an information verification module, an information synchronization module, and an approval process module. Developers first write metadata definition language (MDL) scripts, which describe changes in the database table structure. These MDL scripts are then input into the statement parsing module. The statement parsing module receives MDL script statements submitted by users in the codebase, ensuring compatibility with different versions of relational and non-relational databases. For example, metadata information stored in a database or data lake may include table structures, field definitions, etc.

[0070] Data asset information describes the business value and usage of data, such as its source, purpose, and owner. Data quality information covers aspects like accuracy, completeness, and consistency, used to assess data quality. Data standards information defines the format, structure, and content standards of data, ensuring consistency and comparability. Sensitive data involves personal privacy or data requiring special protection, such as personally identifiable information and financial information. Other metadata information refers to descriptive information related to the data beyond the above, such as creation and modification times.

[0071] Based on natural language processing and syntax analysis technologies, a metadata definition language script statement parsing model is constructed to automatically extract metadata such as table names, field names, data types, and constraints. Named entity recognition technology is used to identify the business meaning of fields, thereby enhancing the semantic understanding of metadata. An information verification module allows developers to perform secondary confirmation of the parsed metadata to ensure accuracy and avoid inconsistencies. After confirmation, the information synchronization module categorizes and synchronizes the metadata information in real time to various metadata information systems, such as the metadata management system, data lake system, and data asset management system, so that data users can access the latest metadata information. Before metadata information synchronization, if metadata changes require approval, such as table structure field adjustments or field name changes, the approval process module will intervene.

[0072] Metadata approvers use this approval process module to review changes, ensuring they comply with the organization's data management requirements and standards. The module can interface with the metadata information system to obtain and update approval process information. Once approved, relevant information is synchronized to all systems in real time, updating metadata information to ensure accuracy and consistency, while improving data governance efficiency.

[0073] See Figure 3The statement parsing module can include a preprocessing submodule, a metadata extraction submodule, and a structured output submodule. The preprocessing submodule performs necessary formatting and cleanup on the metadata definition language script, such as text standardization and comment removal. Next, metadata extraction is performed, identifying and extracting key information such as table names, field names, data types, and constraints from the metadata definition language script. During the model recognition stage, techniques such as named entity recognition are used to determine the business meaning of the fields. Finally, the structured output stage outputs the extracted information in a structured format, facilitating accurate metadata for subsequent information verification, synchronization, and approval processes.

[0074] In an optional embodiment, after parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: when the metadata management system detects that any metadata information meets at least one of the following conditions, it sends the metadata information to the auditing program for verification and auditing; the first condition is used to indicate that the metadata information involves adjustments to table structure fields; the second condition is used to indicate that the metadata information involves changes to field names or changes to data asset information.

[0075] Optionally, the metadata management system has an intelligent detection function when processing metadata information, which can identify whether the metadata information meets specific conditions. When the metadata management system detects that any metadata information meets at least one of the first and second conditions, it can automatically trigger an audit procedure, sending the metadata information to the audit procedure for verification and review. This ensures that critical metadata changes undergo a rigorous audit process before implementation, thereby guaranteeing data consistency and accuracy.

[0076] Optionally, the first condition indicates that the metadata information involves adjustments to table structure fields, which may include adding, deleting, or modifying fields in the table. For example, if developers adjust fields in a table using metadata definition language scripts, such as adding a new field or changing the data type of a field, the metadata management system can identify this change and mark it as requiring review. The second condition indicates that the metadata information involves changes to field names or data asset information. Changes to field names may affect the semantic understanding and use of data, while changes to data asset information may involve important attributes such as data ownership and classification. When these changes are detected, the metadata management system will also send them to the review process so that relevant personnel can conduct detailed verification and review to ensure the rationality and compliance of the changes.

[0077] See Figure 4According to another aspect of the embodiments of this application, a metadata synchronization device is also provided, including: a script acquisition unit 401, a preprocessing unit 402, a parsing operation unit 403, and an information synchronization unit 404.

[0078] The system includes: a script acquisition unit 401 for acquiring a metadata definition language script, which describes changes in the table structure of the database; a preprocessing unit 402 for preprocessing the metadata definition language script to obtain a target metadata definition language script; a parsing operation unit 403 for parsing the target metadata definition language script to obtain at least one type of metadata information, wherein the parsing operation includes: determining the business meaning of each field in the target metadata definition language script through named entity recognition rules, and performing syntax parsing on the target metadata definition language script based on a neural network model; and an information synchronization unit 404 for synchronizing each type of metadata information to the target system according to its type.

[0079] Optionally, the preprocessing unit 402 includes: a text normalization subunit for performing text normalization operations to convert all characters in the metadata definition language script into a uniform form; a comment deletion subunit for performing comment deletion operations to detect and delete comment information in the metadata definition language script; a character processing subunit for performing character processing operations to merge newline characters in the metadata definition language script and remove redundant spaces in the metadata definition language script; a syntax formatting subunit for performing syntax formatting operations to uniformly sort and format field names, data types, and constraints in the metadata definition language script; and a compatibility adjustment subunit for performing compatibility adjustment operations to convert syntax elements in the metadata definition language script into a predefined general format according to the version and syntax differences of different database management systems.

[0080] Optionally, the metadata synchronization device further includes: a statement type detection unit, used to detect the type of each statement in the target metadata definition language script, wherein the statement type is a first type statement, a second type statement, or a third type statement, the first type statement is used to create a table in the database, the second type statement is used to modify a table in the database, and the third type statement is used to delete a table in the database.

[0081] Optionally, the parsing operation unit 403 includes: a name recognition subunit, used to identify the names of each table in the database based on the type of each statement in the target metadata definition language script using regular expressions; a parsing subunit, used to parse the fields related to the database tables in the target metadata definition language script one by one using a recursive descent parsing algorithm and named entity recognition rules to obtain the business meaning of each field; and a first metadata information determination subunit, used to determine the metadata information based on the business meaning of each field.

[0082] Optionally, the parsing operation unit 403 includes: an abstract syntax tree construction subunit, used to construct an abstract syntax tree about the target metadata definition language script using a syntax analysis tool, wherein the abstract syntax tree represents the hierarchical relationship between the table structure, fields and constraints of the target metadata definition language script through a tree structure; and a second metadata information determination subunit, used to determine metadata information based on the abstract syntax tree.

[0083] Optionally, the parsing operation unit 403 includes: a target metadata definition language script acquisition subunit, used to acquire the primary key, non-null value, default value and index information in the target metadata definition language script; a parsing constraint generation subunit, used to generate parsing constraints based on the primary key, non-null value, default value and index information; and a parsing operation subunit, used to perform parsing operations on the target metadata definition language script according to the parsing constraints to obtain at least one metadata information.

[0084] Optionally, the metadata synchronization device further includes: a column operation instruction parsing unit, used to parse the column operation instructions corresponding to the second type of statement when the target metadata definition language script includes a second type of statement, wherein the column operation instruction is used to represent the instruction to add or modify a column in a table in the database.

[0085] Optionally, the metadata synchronization device further includes: an information sending unit for sending the obtained metadata information to the user interface; and a verification result determination unit for determining the verification result of the metadata information based on the user instruction information returned by the user interface.

[0086] Optionally, the metadata synchronization device further includes: a verification and auditing unit, used to send the metadata information to the auditing program for verification and auditing when it is detected that any metadata information meets at least one of the following conditions; wherein, the first condition is used to indicate that the metadata information involves adjustments to table structure fields; and the second condition is used to indicate that the metadata information involves changes to field names or changes to data asset information.

[0087] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described metadata synchronization method.

[0088] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors perform the above-described metadata synchronization method.

[0089] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that implement the above-described metadata synchronization method when executed by a processor.

[0090] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0091] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0096] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A metadata synchronization method, characterized in that, include: Obtain a metadata definition language script, wherein the metadata definition language script is used to describe changes in the table structure of the database; The metadata definition language script is preprocessed to obtain the target metadata definition language script; The target metadata definition language script is parsed to obtain at least one metadata information, wherein the parsing operation includes: determining the business meaning of each field in the target metadata definition language script through named entity recognition rules, and performing syntax parsing on the target metadata definition language script based on a neural network model; Each type of metadata information is synchronized to the target system according to its type.

2. The metadata synchronization method according to claim 1, characterized in that, The preprocessing includes at least one of the following operations: Text standardization operations are used to convert all characters in the metadata definition language script into a uniform form; The comment deletion operation is used to detect and delete comment information in the metadata definition language script; Character processing operations are used to merge newline characters in the metadata definition language script and remove redundant spaces in the metadata definition language script. Syntax formatting operations are used to uniformly sort and format field names, data types, and constraints in the metadata definition language script; The compatibility adjustment operation is used to convert the syntax elements in the metadata definition language script into a predefined general format based on the version and syntax differences of different database management systems.

3. The metadata synchronization method according to claim 1, characterized in that, Before parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: The type of each statement in the target metadata definition language script is detected, wherein the statement type is a first type statement, a second type statement, or a third type statement. The first type statement is used to create a table in the database, the second type statement is used to modify a table in the database, and the third type statement is used to delete a table in the database.

4. The metadata synchronization method according to claim 3, characterized in that, The target metadata definition language script is parsed to obtain at least one metadata information, including: Based on the type of each statement in the target metadata definition language script, regular expressions are used to identify the names of each table in the database. Using a recursive descent parsing algorithm, the fields related to the database table in the target metadata definition language script are parsed one by one according to the named entity recognition rules to obtain the business meaning of each field; The metadata information is determined based on the business meaning of each field.

5. The metadata synchronization method according to claim 1, characterized in that, The target metadata definition language script is parsed to obtain at least one metadata information, including: An abstract syntax tree for the target metadata definition language script is constructed using a syntax analysis tool. The abstract syntax tree represents the hierarchical relationship between the table structure, fields, and constraints of the target metadata definition language script through a tree structure. The metadata information is determined based on the abstract syntax tree.

6. The metadata synchronization method according to claim 1, characterized in that, The target metadata definition language script is parsed to obtain at least one metadata information, including: Obtain the primary key, non-null value, default value, and index information from the target metadata definition language script; Generate parsed constraints based on the primary key, non-null values, default values, and index information; The target metadata definition language script is parsed according to the parsing constraints to obtain at least one metadata information.

7. The metadata synchronization method according to claim 3, characterized in that, The metadata synchronization method further includes: When the target metadata definition language script includes statements of the second type, the column operation instructions corresponding to the statements of the second type are parsed, wherein the column operation instructions are used to represent instructions to add or modify columns in a table in the database.

8. The metadata synchronization method according to claim 1, characterized in that, After parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: The obtained metadata information is sent to the user interface. The verification result of the metadata information is determined based on the user instruction information returned by the user interaction interface.

9. The metadata synchronization method according to claim 1, characterized in that, After parsing the target metadata definition language script to obtain at least one metadata information, the metadata synchronization method further includes: When any metadata information is detected to meet at least one of the following conditions, the metadata information is sent to the auditing program for verification and auditing. The first condition is used to characterize metadata information involving adjustments to table structure fields; The second condition is used to characterize metadata information involving changes in field names or data asset information.

10. A metadata synchronization device, characterized in that, include: The script acquisition unit is used to acquire metadata definition language scripts, wherein the metadata definition language scripts are used to describe changes in the table structure of the database; The preprocessing unit is used to preprocess the metadata definition language script to obtain the target metadata definition language script; A parsing operation unit is used to perform parsing operations on the target metadata definition language script to obtain at least one metadata information, wherein the parsing operation includes: determining the business meaning of each field in the target metadata definition language script through named entity recognition rules, and performing syntax parsing on the target metadata definition language script based on a neural network model; The information synchronization unit is used to synchronize each type of metadata information to the target system according to the type of each type of metadata information.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device in which the computer-readable storage medium is located performs the metadata synchronization method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the metadata synchronization method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the metadata synchronization method of any one of claims 1 to 9.

Images