Metadata management method, system and equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a metadata management method, system and device and a storage medium, and the method comprises the steps: extracting feature information of metadata, and generating attribute information based on the feature information; distributing a key prefix for metadata based on the attribute information, and storing the metadata to a pre-distribution logic interval of the key prefix in a key value pair form; respectively calculating index values of attribute information of the metadata in each logic interval by utilizing a preset index rule; and confirming that the index value of the logic interval exceeds a set index threshold value, and generating metadata exception early warning. According to the method and the device, efficient management of the metadata is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a metadata management method, system, device and storage medium. Background Art

[0002] Metadata, also known as intermediary data or relay data, is data about data. It is mainly information describing data properties, used to support functions such as indicating storage location, historical data, resource search, and file records. Metadata is a kind of electronic catalog. In order to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval.

[0003] At present, the financial industry's control over stock metadata is still in a passive vicious circle that basically relies on the professionalism of developers. That is, the current metadata governance is heavily dependent on the rules set by developers. However, due to the complexity and diversity of metadata, it is not necessarily adapted to the original developed rules, so the difficulty of metadata governance has encountered a bottleneck. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a metadata management method, system, device and storage medium to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a metadata management method, comprising: Extracting feature information of metadata, and generating attribute information based on the feature information; assigning a key prefix to the metadata based on the attribute information, and saving the metadata in a pre-assigned logical interval of the key prefix in the form of a key-value pair; Calculate the index value of the attribute information of the metadata in each logical interval respectively by using the preset index rule; Confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate a metadata abnormality warning.

[0006] In an optional implementation, extracting feature information of metadata and generating attribute information based on the feature information includes: The entity naming model is used to extract characteristic information from the ledger and metadata directory, wherein the characteristic information includes attribution source information, technical information, and resource information; the attribution source information includes attribution application system, business management identity information, and technical management identity information; the technical information includes database information, mode information, table-level information, field-level information, and index information; the resource information includes data resource attribution identity information, data resource status, and attribution development platform information; Set the attribution source information as the management attribute of the metadata, set the technical information as the technical attribute of the metadata, and set the resource information as the resource archiving attribute of the metadata; According to the technical information and pre-set technical standard parameters, standard comparison information is generated according to whether the technical information conforms to the corresponding technical standard parameters, and the standard comparison information is set as a data matching attribute of the metadata; the technical standard parameters include a mapping standard name corresponding to a combination of a table name and a field name, a standard data type, a standard data length, a standard data precision and a standard code item.

[0007] In an optional implementation, assigning a key prefix to the metadata based on the attribute information, and saving the metadata in the form of a key-value pair to a pre-assigned logical interval of the key prefix includes: Determine the data type of the metadata according to the attribute information; assigning a key prefix to the metadata according to a correspondence between the data type and the key prefix and the determined data type; Assign a logical interval to each key prefix in advance, and store the data type, key prefix, and logical interval address of the metadata in a configuration file; The metadata is serialized and stored in the corresponding logical interval.

[0008] In an optional implementation, the index value of the attribute information of the metadata in each logical interval is calculated respectively using a preset index rule, including: Set the weights of each parameter item of the data benchmarking attribute and calculate the weighted sum to obtain the data item-level standard compliance; Perform weighted summation based on the preset data item weights and item-level standard compliance to obtain table-level standard compliance; A weighted sum is performed according to the preset table weight and the table-level standard compliance to obtain the application-level standard compliance.

[0009] In an optional implementation, confirming that the index value of the logic interval exceeds a set index threshold and generating a metadata abnormality warning includes: A threshold combination is set for each logical interval respectively, wherein the threshold combination includes a data item level standard compliance threshold, a table level standard compliance threshold and an application level standard compliance threshold; Compare the data item-level standard compliance, table-level standard compliance and application-level standard compliance within the logical interval with the corresponding thresholds. If they are lower than the corresponding thresholds, a corresponding abnormal warning is generated; The One-Class SVM model is used to analyze the boundary value of each logical interval and determine the abnormal metadata based on the boundary value.

[0010] In a second aspect, the present invention provides a metadata management system, comprising: An attribute generation module, used to extract feature information of metadata and generate attribute information based on the feature information; A data storage module, configured to assign a key prefix to the metadata based on the attribute information, and save the metadata in a pre-assigned logical interval of the key prefix in the form of a key-value pair; An index calculation module, used to calculate the index value of the attribute information of the metadata in each logical interval respectively using a preset index rule; The abnormal warning module is used to confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate a metadata abnormal warning.

[0011] In an optional implementation, the attribute generation module includes: A feature extraction unit is used to extract feature information from the ledger and metadata directory using an entity naming model, wherein the feature information includes attribution source information, technical information, and resource information; the attribution source information includes attribution application system, business management identity information, and technical management identity information; the technical information includes database information, mode information, table-level information, field-level information, and index information; the resource information includes data resource attribution identity information, data resource status, and attribution development platform information; A first setting unit is used to set the attribution source information as the management attribute of the metadata, set the technical information as the technical attribute of the metadata, and set the resource information as the resource archiving attribute of the metadata; The second setting unit is used to generate standard comparison information according to the technical information and pre-set technical standard parameters, and whether the technical information conforms to the corresponding technical standard parameters, and set the standard comparison information as a data matching attribute of the metadata; the technical standard parameters include a mapping standard name corresponding to a combination of a table name and a field name, a standard data type, a standard data length, a standard data precision and a standard code item.

[0012] In an optional implementation, the data storage module includes: A data classification unit, used to determine the data type of the metadata according to the attribute information; an identification allocation unit, configured to allocate a key prefix to the metadata according to a correspondence between the data type and the key prefix and the determined data type; A configuration storage unit is used to pre-allocate a logical interval for each key prefix, and store the data type, key prefix and logical interval address of the metadata in a configuration file; The encoding storage unit is used to serialize the metadata and store the processed metadata in a corresponding logical interval.

[0013] In a third aspect, a device is provided, comprising: A memory for storing a metadata management program; The processor is used to implement the steps of the metadata management method provided in the first aspect when executing the metadata management program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, on which a metadata management program is stored. When the metadata management program is executed by a processor, the steps of the metadata management method provided in the first aspect are implemented.

[0015] The beneficial effects of the present invention are that the metadata management method, system, device and storage medium provided by the present invention realize efficient management of metadata by generating attributes for metadata and classifying and storing them based on the attributes, while performing indicator calculation and anomaly detection on metadata in different storage areas.

[0016] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0019] Figure 2 The figure is a schematic diagram of a scenario of a method according to an embodiment of the present invention.

[0020] Figure 3 It is a detailed scenario schematic diagram of a method according to an embodiment of the present invention.

[0021] Figure 4 is a schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 5 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0025] The metadata management method provided by the embodiment of the present invention is executed by a computer device, and accordingly, the metadata management system runs in the computer device.

[0026] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a metadata management system. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.

[0027] like Figure 1 As shown, the method includes: S1, extracting feature information of metadata, and generating attribute information based on the feature information; S2, assigning a key prefix to the metadata based on the attribute information, and saving the metadata in the form of a key-value pair to a pre-assigned logical interval of the key prefix; S3, using preset index rules to calculate the index values ​​of the attribute information of the metadata in each logical interval; S4, confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate a metadata abnormality warning.

[0028] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0029] S101. Use the entity naming model to extract feature information from the ledger and metadata directory, the feature information includes source information, technical information, and resource information; the source information includes the application system, business management identity information, and technical management identity information; the technical information includes database information, schema information, table-level information, field-level information, and index information; the resource information includes data resource identity information, data resource status, and development platform information.

[0030] The entity naming model is an intelligent model built on advanced natural language processing and machine learning technologies. It can accurately identify and extract various entity information from text data. In the data processing process, the ledger and metadata directory are used as input data sources to give full play to the powerful information extraction capabilities of the entity naming model.

[0031] Through the operation of the entity naming model, multi-dimensional feature information was successfully extracted from the ledger and metadata catalog. This feature information covers important aspects such as attribution source information, technical information, and resource information. Each type of information provides a unique perspective and key clues for in-depth understanding of the data.

[0032] First, let's look at the attribution source information, which is the core element to understanding the source and ownership of data. The attribution source information is further subdivided into the attribution application system, business management identity information, and technical management identity information. The attribution application system clarifies the specific application or system from which the data originated, which is important for tracking the background and business process of the data. For example, a batch of data may come from the company's customer relationship management system (CRM). By clarifying the attribution application system, it is possible to clearly understand the close connection between this data and the customer management business.

[0033] Business management identity information involves relevant information about business personnel who are responsible for data management and decision-making. This information includes the name, position, and department of the business person in charge, who plays a key role in data business planning, usage decisions, and coordination with other business links. Knowing business management identity information helps to better communicate and collaborate with relevant business personnel during the data analysis and application process, ensuring that the use of data meets business needs and strategic directions.

[0034] Technical management identity information focuses on the department information that manages and maintains data from a technical perspective. They are responsible for technical work such as data storage, processing, and security assurance, and play a decisive role in the quality and availability of data. Clarifying technical management identity information enables the relevant responsible persons to be quickly located when encountering data technology problems, solve problems in a timely manner, and ensure the normal circulation and use of data.

[0035] Next is the technical information, which provides an in-depth look at the technical composition and characteristics of the data. The technical information includes database information, schema information, table-level information, field-level information, and index information.

[0036] Database information covers key content such as database type (such as relational database, non-relational database), version, storage location, etc. Different types of databases have their own unique advantages and applicable scenarios. Understanding database information helps to select appropriate database management tools and technical means based on the characteristics of the data and business needs. For example, for data with a high degree of structure and strict transaction processing requirements, a relational database may be a better choice; while for processing massive unstructured data, a non-relational database is more advantageous.

[0037] Schema information describes the organization and structure of data in the database, including the database architecture design, the relationship between tables, etc. Clear schema information provides important guidance for data query, analysis and maintenance, enabling accurate understanding of the logical relationship between data and avoiding errors during data operations.

[0038] Table-level information provides detailed information such as the name, description, creation time, and modification time of each table in the database. This information helps you quickly understand the purpose and life cycle of each table and determine the timeliness and availability of data. For example, by viewing the creation time and modification time of a table, you can evaluate the update frequency of the data and determine whether the data needs to be updated or maintained in a timely manner.

[0039] Field-level information goes deep into each field in the table, and describes in detail the field's name, data type, length, whether it is empty, and other attributes. Field-level information is crucial for data entry, query, and analysis. It determines the format and value range of the data and ensures the accuracy and consistency of the data. For example, when entering data, you can set corresponding validation rules based on the attributes of the field to prevent the entry of incorrect data.

[0040] Index information is a data structure established to improve data query efficiency. Through the index information extracted by the entity naming model, you can understand which fields in the database have indexes, as well as the types and usage of the indexes. Reasonable index design can significantly improve the speed of data query and optimize database performance.

[0041] Finally, there is resource information, which comprehensively reflects the data resource in terms of ownership, status, and development platform. Resource information includes data resource ownership identity information, data resource status, and ownership development platform information.

[0042] Data resource ownership identity information clarifies the owner or ownership team of the data resource, which is of great significance for data authority management and responsibility division. Only by clarifying the ownership identity of the data can the legal use and protection of the data be ensured, and the risk of data leakage and abuse be avoided.

[0043] The data resource status describes the current state of the data resource, such as whether it is available, whether it is being updated, whether there are errors, etc. Timely understanding of the data resource status helps to reasonably arrange the use and processing tasks of data and avoid business interruptions or errors caused by abnormal data status. For example, if the data resource is being updated, you can pause the related data query operation and wait until the update is completed before continuing to obtain the latest and accurate data.

[0044] The development platform information indicates the development platform or technical framework that the data resources rely on. Different development platforms have different characteristics and functions. Understanding the development platform information helps to better understand the development background and technical environment of the data, and provides an important reference for subsequent data migration, upgrade or integration.

[0045] In summary, by using the entity naming model to extract multi-dimensional feature information covering source information, technical information and resource information from the ledger and metadata catalog, it is possible to build a comprehensive and detailed data portrait, provide a solid foundation for data management, analysis and application, and help enterprises make more informed decisions in the digital age.

[0046] S102. Set the attribution source information as the management attribute of the metadata, set the technical information as the technical attribute of the metadata, and set the resource information as the resource archiving attribute of the metadata.

[0047] The following are some examples of properties: Management properties (example): Attribution application system: private customer information management; Business management department: Retail Finance Department; Technical Management Department: R&D Department 1.

[0048] Technical properties (example): Database information: GoldenDb; Mode information: NCIM; Table level information (partial): Table English name: TBPC1090; Table Chinese name: Customer financial information; Creation date: November 12, 2021; Field level information: Field English name: CST_ID; Field Chinese name: Customer Number; Field type: fixed-length character class; Field length: 18; Whether to allow empty space: No; Is it primary key: yes; Index information: Index fields.

[0049] Data resource archive information (example): Department to which data resources belong: Retail Finance Department; Data resource status: enabled; Development platform information: Business management department information: Retail Finance Department; Technical management department information: R&D Department 1; Data Management Department Information: Data Governance Team.

[0050] S103. Based on the technical information and pre-set technical standard parameters, standard comparison information is generated according to whether the technical information conforms to the corresponding technical standard parameters, and the standard comparison information is set as the data matching attribute of the metadata; the technical standard parameters include a mapping standard name corresponding to the combination of the table name and the field name, a standard data type, a standard data length, a standard data precision and a standard code item.

[0051] For example, data benchmarking attributes (example): Table name: TBPC1090; Field name: CST_ID; Mapping standard name: Customer Number; Data type benchmarking result: compliant; Data length benchmarking result: compliant; Data accuracy benchmarking results: None; Code item benchmarking results: None.

[0052] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0053] S201. Determine the data type of the metadata according to the attribute information.

[0054] In the data processing process, after obtaining the attribute information of metadata, the first task is to accurately determine the data type of metadata. Attribute information contains various feature descriptions of metadata, which are the key basis for determining the data type.

[0055] For example, for metadata describing customer information, its attribute information may include customer name, age, contact information, purchase history, etc. Different data types can be analyzed from these attributes. Customer name belongs to the text type because it is descriptive information composed of characters; age is usually a numeric type, used to represent a specific quantity; contact information may be a phone number or an email address, which is text in form but has a specific format and purpose, and can also be regarded as a special text type; and purchase records may involve different data type combinations such as date and amount.

[0056] In actual operation, a series of detailed judgment rules will be formulated. If the attribute information contains clear numerical ranges, measurement units and other numerical-related features, then the metadata is likely to be of numerical type, such as sales, inventory quantity, etc. For attributes that contain character sequences and have no obvious numerical features, if they are used to describe things, events or concepts, they are usually judged as text types, such as product names, project descriptions, etc.

[0057] In addition, for some information with specific formats and meanings, such as information in date and time format (such as "2023-10-05 14:30:00"), it will be determined as a date and time type; while for information that represents logical judgments such as "yes" or "no", "true" or "false", it will be classified as a Boolean type. Through these detailed analysis and judgment rules, the data type of metadata can be accurately determined based on attribute information.

[0058] S202. Allocate a key prefix to the metadata according to the correspondence between the data type and the key prefix and the determined data type.

[0059] After determining the data type of the metadata, you need to assign a suitable key prefix to the metadata based on the correspondence between the data type and the key prefix. This correspondence is pre-set and is intended to classify and identify different types of metadata through key prefixes, so that data can be managed and retrieved more efficiently in the future.

[0060] For example, set the data key prefix of the numeric type to "num_", the data key prefix of the text type to "txt_", the data key prefix of the date and time type to "dt_", the data key prefix of the Boolean type to "bool_", and so on.

[0061] Assuming that the data type of a metadata is determined to be text type after step S201, then according to the above correspondence, "txt_" will be assigned as the key prefix for this metadata. If it is a numeric type metadata indicating the age of a customer, it will be assigned the key prefix "num_".

[0062] This correspondence is not static, and enterprises can flexibly adjust it according to their own data management needs and business logic. For example, in a specific business scenario, for numeric metadata involving financial data, a key prefix of "fin_num_" may be set separately to better distinguish and manage financial-related data. By reasonably setting the correspondence between data types and key prefixes, and strictly assigning key prefixes to metadata according to this relationship, it can ensure that data classification is clear and orderly.

[0063] S203. Allocate a logical interval for each key prefix in advance, and store the data type, key prefix and logical interval address of the metadata in a configuration file.

[0064] In order to store and manage metadata more efficiently, it is necessary to pre-allocate a specific logical interval for each key prefix. A logical interval is a specific area divided in the data storage system to store metadata with the same key prefix.

[0065] For example, for the key prefix "num_", the logical interval [1000 - 1999] may be assigned; for the key prefix "txt_", the logical interval [2000 - 2999] may be assigned; the logical interval corresponding to the key prefix "dt_" may be [5000 -3999], etc. The division of these logical intervals is based on multiple factors such as data volume estimation, storage performance optimization, and data management convenience.

[0066] After assigning logical intervals to each key prefix, the metadata data type, key prefix, and corresponding logical interval address need to be stored in the configuration file. The configuration file is an important data management tool that records various key information in the data processing process, making it easy for the system to quickly obtain and use it in subsequent operations.

[0067] The configuration file can be in a common text format, such as JSON or XML. For example: { "metadata_config": [ { "data_type": "Numeric type", "key_prefix": "num_", "logical_interval": [1000, 1999]}, { "data_type": "Text type", "key_prefix": "txt_", "logical_interval": [2000, 2999]}, { "data_type": "Date time type", "key_prefix": "dt_", "logical_interval": [5000, 3999]} ]}.

[0068] By storing this information in a configuration file, when the system needs to query or process a certain piece of metadata, it can quickly locate the data type, key prefix, and storage logical range address to which the metadata belongs, greatly improving the efficiency and accuracy of data processing.

[0069] S204. Serialize the metadata and store the processed metadata in the corresponding logical range.

[0070] After completing the previous steps, in order to store the metadata in the corresponding logical range, it is necessary to serialize the metadata first. Serialization is the process of converting a data object into a format that can be transmitted over a network or stored in a file. It converts complex data structures into byte sequences for easy storage and transmission.

[0071] For example, for a metadata object containing multiple attributes, such as customer information metadata (including name, age, contact information, etc.), it exists in a specific object structure in memory. But when storing it on a physical medium (such as a hard disk), it needs to be converted into a format suitable for storage. Common serialization formats include JSON, XML, ProtocolBuffers, etc.

[0072] Suppose JSON format is selected for serialization processing. For a customer information metadata object: customer = { "name": "Zhang San", "age": 30, "contact": "138xxxxxxxx"}.

[0073] After serialization processing, it will be converted into the following JSON string: {"name": "Zhang San", "age": 30, "contact": "138xxxxxxxx"}.

[0074] After completing the serialization processing, the system will find the corresponding logical range according to the previously determined key prefix and store the processed metadata in that logical range. For example, if the data type of this customer information metadata is text type and the key prefix is “txt_”, then the system will store the serialized metadata in the logical range [2000 - 2999] allocated in advance for the “txt_” key prefix.

[0075] In an embodiment of the present invention, based on step S3, a possible embodiment will be given below to non - restrictively elaborate on its specific implementation scheme.

[0076] This step generates indicators in batch processing mode and creates multiple threads, which synchronously process metadata of multiple logical areas.

[0077] S301. Set the weight of each parameter item of the data benchmarking attribute, and calculate the weighted sum to obtain the data item-level standard compliance.

[0078] Composite indicators are aggregated from multiple basic indicators such as whether the data type is consistent, whether the data length is consistent, whether the data precision is consistent, and the code item consistency. Each basic indicator needs to consider the basic indicator values ​​in different database tables in the system, and the weight ratio in the entire data table is different according to whether the data item is required, whether it is an index, whether it is a partition key, etc. In addition, different weights are set in the system according to the importance of each table in the application system, the amount of data, etc., and finally the data standard compliance of the application system is calculated and aggregated.

[0079] Set the data type to match , the weight is , whether the data length matches , the weight is , whether the data accuracy is consistent with , the weight is , the code item matches , the weight is , then each basic indicator value is , the weight value is , then the calculated data item-level data standard compliance is .

[0080] S302. Perform weighted summation based on preset data item weights and item-level standard compliance to obtain table-level standard compliance.

[0081] The weight of common data items is determined by the different requirements of management attributes, business attributes, and technical attributes of the data items. Management attributes include the line to which they belong, the business development and operation team to which they belong, business attributes include the data model theme to which they belong, and technical attributes include whether the data item is a required input, index, or partition key. The field weight of the data item is Z. i , then the table-level data standard compliance of the table can be calculated as .

[0082] S303. Perform weighted summation based on the preset table weight and the table-level standard compliance to obtain the application-level standard compliance.

[0083] Assume that the weight of the common table in the system is Q iIf the table is a parameter table, business table, or transaction table, different fixed values ​​are adjusted up. If the table is a statistical table, intermediate table, or flow table, different fixed values ​​are adjusted down. Then the weight of the table in the application system is Q i , then the calculated application-level data standard compliance of the application system is .

[0084] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0085] S401. Set a threshold combination for each logical interval respectively, wherein the threshold combination includes a data item-level standard compliance threshold, a table-level standard compliance threshold, and an application-level standard compliance threshold.

[0086] In the complex system of data management and quality monitoring, in order to accurately identify and process possible abnormal data, it is necessary to set detailed thresholds for each logical interval. As the basic unit of data storage and management, the logical interval carries metadata of different types and purposes.

[0087] For each logical interval, a comprehensive threshold combination is constructed, which covers the standard compliance thresholds at three different levels: data item level, table level, and application level.

[0088] Data item level standard compliance threshold: Data item is the most basic unit of data, such as each field value in a database table. The data item level standard compliance threshold is used to measure the degree of match between a single data item and a pre-set standard. This standard can be the format, value range, data type, etc. of the data. For example, for a data item representing age, its value range is set to be between 0 and 120, and the data type is an integer. Then, the data item level standard compliance threshold is used to determine whether the age data item is within this reasonable range and whether it meets the integer type requirements. If the degree to which the data item meets these standards is lower than the set threshold, it may mean that there is a potential problem with the data. The specific threshold setting will be determined based on the importance of the data, business needs, and analysis of historical data. For key business data, a higher threshold may be set to ensure the accuracy and reliability of the data; while for some auxiliary data, the threshold can be relatively loose.

[0089] Table-level standard compliance threshold: The table-level standard compliance threshold focuses on the overall quality of the entire data table. It not only considers the situation of a single data item, but also involves aspects such as the integrity and consistency of the data in the table and the logical relationship between the data. For example, in a customer information table, each customer's unique identification field (such as customer ID) should be unique, without duplicate values; there may be certain correlations between the fields, such as the date of birth and age fields should match each other. The table-level standard compliance threshold is used to evaluate the performance of the entire table in these aspects. This threshold is set by considering multiple factors, such as the missing rate of data in the table, the degree of deviation in data consistency, etc. If the table-level standard compliance is lower than the set threshold, it means that there may be major problems with the data quality of the table, which requires further in-depth analysis and processing.

[0090] Application-level standard compliance threshold: The application-level standard compliance threshold is set from a higher level, that is, from the perspective of data in actual application. It takes into account the availability, effectiveness and support of data for the entire application system and the degree of business process. For example, in an e-commerce order processing system, the timely update and accuracy of order data are crucial for business links such as inventory management and delivery arrangements. The application-level standard compliance threshold is used to measure the performance of data in meeting these business needs. This may involve factors such as the frequency of data updates and the degree of match between data and business rules. If the application-level standard compliance is lower than the threshold, it may have a serious impact on the normal operation of the entire application system and the smooth development of business.

[0091] By carefully setting the threshold combinations of these three levels for each logical interval, it is possible to comprehensively monitor and evaluate the data from different angles, providing an accurate basis for subsequent anomaly detection and processing.

[0092] S402. Compare the data item-level standard compliance, table-level standard compliance and application-level standard compliance within the logic interval with corresponding thresholds, and generate corresponding abnormal warnings if they are lower than the corresponding thresholds.

[0093] After completing the threshold setting for each logical interval, the next key step is to monitor and evaluate the data within the logical interval in real time. It is necessary to calculate the data item-level standard compliance, table-level standard compliance, and application-level standard compliance in each logical interval, and compare these compliances with the corresponding thresholds set previously.

[0094] Comparison of data item-level standard compliance: For each data item within the logical interval, its compliance is calculated according to the pre-set data item standard. For example, for a data item in date format, it will be checked whether it conforms to the specified date format (such as "YYYY - MM - DD"). If it conforms, the format compliance is 100%; then check whether its value is within a reasonable time range, such as whether it is within the business time span allowed by the system. Combine these factors to calculate the overall standard compliance of the data item. Then, compare this compliance with the data item-level standard compliance threshold. If the data item's compliance is lower than the threshold, the system will immediately generate a data item-level abnormality warning. This warning will record in detail the name of the data item, the name of the table where it is located, the specific non-compliance situation, and the possible impact on the business, so that relevant personnel can conduct timely investigation and processing.

[0095] Comparison of table-level standard compliance: Based on the data item-level compliance assessment, the entire table is further comprehensively evaluated to calculate the table-level standard compliance. This requires consideration of multiple factors, such as data integrity (whether there are a large number of missing values), data consistency (whether the logical relationship between different fields is correct), and data accuracy (whether the data is consistent with the actual situation). Through a series of complex algorithms and rules, the standard compliance of the table is obtained. Then, this compliance is compared with the table-level standard compliance threshold. If the table-level standard compliance is lower than the threshold, the system will generate a table-level abnormality warning. This warning will include information such as the name of the table, the main non-compliant aspects, and the possible impact on related business modules. Based on this warning, relevant personnel can conduct a comprehensive review and repair of the data of the entire table to ensure the quality and availability of the data.

[0096] Comparison of application-level standard compliance: Finally, from the perspective of the application system as a whole, calculate the application-level standard compliance. This requires considering the use of data in various business processes and the degree to which the data supports business goals. For example, in a financial report generation system, it will be checked whether the financial data can generate reports in a timely and accurate manner, and whether the report data complies with financial regulations and business requirements. By evaluating these aspects, the application-level standard compliance is obtained. It is compared with the application-level standard compliance threshold. If it is lower than the threshold, the system will generate an application-level anomaly warning. This warning will emphasize the potential impact of the anomaly on the entire application system and business operations, prompting the relevant teams to take quick action to ensure the normal operation of the business.

[0097] Through this detailed comparison and early warning mechanism, problems existing in data at different levels can be discovered in a timely manner, providing strong guarantees for improving data quality and the stable development of the business.

[0098] The following are examples of warning rules: Definition of warning rules: Issue warnings on deviations from data standards for different application systems in the user acceptance test environment.

[0099] Warning level: medium; Warning frequency: every Thursday; Warning rule processing logic: Take the data standard deviation of different application systems in the user acceptance test environment, and give no warning for deviations in the range of -5% to 0%. Give a blue prompt for deviations in the range of -5% to -10%. Give a yellow warning for deviations in the range of -15% to -10%. Give a red warning for deviations greater than -15%.

[0100] Supplementary explanation: The data standard deviation index describes the deviation between the application-level data standard compliance calculated by each system and the application system benchmark value, where the application system benchmark value is a fixed value set according to the application system's business load and the importance of the application system, and is adjusted quarterly. The specific calculation formula is data standard deviation (ρ) = (application-level data standard compliance - application system benchmark value) / application system benchmark value * 100%.

[0101] S403. Analyze the boundary value of each logical interval using the One-Class SVM model, and determine abnormal metadata based on the boundary value.

[0102] After generating anomaly warnings through threshold comparison, the One-Class SVM (one-class support vector machine) model is introduced to more accurately identify and locate abnormal metadata. One-Class SVM is a powerful machine learning algorithm that is particularly suitable for finding the boundaries of normal data in a data set, thereby identifying abnormal data that deviates from the normal range.

[0103] The core idea of ​​the One-Class SVM model is to find an optimal hyperplane in high-dimensional space to contain most of the normal data in an area. The boundary of this area is the boundary of the normal data. The model determines the distribution range of normal data by maximizing the distance between normal data and this hyperplane. During the training process, the model only uses normal data for learning, thereby constructing a model that can describe the characteristics of normal data.

[0104] For each logical interval, the metadata in it is used as input and analyzed using the One-Class SVM model. The model will find the boundary values ​​of normal data in the feature space based on the input metadata features. These boundary values ​​represent the feature range of normal metadata in the logical interval. For example, in a logical interval containing customer transaction amounts, the model will analyze the distribution of transaction amounts and determine the upper and lower limits of normal transaction amounts, which are the boundary values.

[0105] After obtaining the boundary values ​​of each logical interval, abnormal metadata can be determined based on these boundary values. For each metadata in the logical interval, its features are compared with the boundary values. If the features of the metadata exceed the range defined by the boundary values, then this metadata is determined to be abnormal metadata. For example, if the transaction amount of a customer far exceeds the upper limit of the normal transaction amount determined by the One-Class SVM model, then the metadata corresponding to this transaction record will be marked as abnormal.

[0106] By using the One-Class SVM model to analyze boundary values ​​to determine abnormal metadata, it is possible to more accurately identify abnormal data points hidden in large amounts of data, providing precise targets for further data processing and problem solving. This method not only improves the accuracy of anomaly detection, but also helps to gain a deeper understanding of the inherent characteristics and distribution patterns of the data, providing strong support for optimizing data management strategies.

[0107] In an application scenario, please refer to Figure 2 ,Metadata management mainly includes three parts: metadata basic ,control, metadata control improvement, and scheduling platform integration.

[0108] The basic metadata management and control includes modules such as metadata attribute definition, metadata collection, metadata catalog, and metadata benchmarking.

[0109] Metadata attribute definition is to describe metadata in detail as a whole, divide the description of metadata into multiple attributes, and refine the process of attribute definition. At present, the various definition attributes of our bank's metadata are mainly divided into several categories of information, such as management information, technical information, data standard benchmarking information, and data resource archiving information. Management information mainly includes the source system to which this type of metadata belongs and the corresponding business management department, technical department and corresponding person in charge of the system; technical information includes database information, model information, table-level information, field-level information, index information, etc.; data standard benchmarking information mainly includes detailed information on the results formed after the consistency comparison and verification of metadata and data standards, including whether the name is consistent, whether the data type is consistent, whether the data length is consistent, whether the data precision is consistent, whether the code item is consistent with the data standard, etc.; data resource archiving information mainly includes the classification and attribution of this type of metadata to data resources.

[0110] Among them, management information mainly comes from ledger collection and maintenance, technical information mainly comes from the metadata directory formed after the metadata collection function of the metadata basic management and control module is executed, and data standard benchmarking information comes from the comparison result information after the collected metadata is benchmarked with the enterprise-level data standard, including whether a certain field in a certain table should be benchmarked and the benchmarking results. The benchmarking results cover whether the data type is consistent, whether the data length and data accuracy are consistent, and other information.

[0111] Please refer to Figure 3 , metadata control and improvement mainly includes three modules: metadata indicator management, metadata warning management, and metadata comprehensive autonomous analysis. Among them, metadata indicator management functions mainly include metadata indicator application, metadata indicator maintenance, metadata indicator review, metadata indicator processing, metadata indicator scheduling and other functions; metadata warning management mainly includes metadata warning rule maintenance, metadata warning rule review, metadata warning rule processing, metadata warning rule configuration scheduling, metadata warning signal disposal and other functions; metadata comprehensive autonomous analysis module mainly includes autonomous query (report) maintenance, autonomous query publishing, autonomous report publishing, large screen data assembly, large screen data publishing, autonomous publishing review and other functions.

[0112] The metadata indicator application function is to initiate the definition and application process of metadata indicators. Through this function, the metadata indicator number, indicator name, indicator classification, indicator introduction, indicator type, indicator business rules, indicator processing rules and other information are maintained to complete the definition of metadata indicators. The indicator processing rules are carried out by self-selection and input of logical operators. First, the selected metadata attribute definition is selected, and then the processing logic of the metadata indicator is formed by self-combining, arranging or logically processing the formula.

[0113] Metadata indicator application process, for example: "Fill in basic indicator information" -> "Basic attributes", fill in "Indicator name", "Indicator alias", "Indicator classification", "Indicator unit", and then click "Next". "Fill in basic indicator information" -> "Business attributes", fill in "Business definition", "Business purpose", "Business rules", and then click "Next". "Fill in basic indicator information" -> "Technical attributes", fill in "Data type", "Processing rules", and then click "Next". "Fill in basic indicator information" -> "Management attributes", fill in "Contact person of the competent department" and "Demand proposer", and then click "Next". "Fill in basic indicator information" -> "Data labeling", fill in "Is it a Class A indicator", "Whether it complies with data standards", "Whether it is a data standard", "Data standard number", "Data security classification", "Data security classification", "Adjustment suggestions", "Data desensitization rules", and then click "Next". "Fill in basic indicator information" -> "Quality requirements", fill in "Business quality requirements", "Technical quality requirements", "Verification script", and then click "Next".

[0114] The metadata indicator maintenance function is the process of modifying the definition of metadata indicators that have not been reviewed or have been reviewed, and re-entering the review process after modification. The metadata indicator maintenance function can be used to modify the metadata indicator name, indicator classification, indicator introduction, indicator type, indicator business rules, and indicator processing rules. The metadata indicator number cannot be modified. Metadata indicators that have been included in the metadata indicator processing task cannot be modified.

[0115] The metadata indicator review function is a function for reviewing the initiated metadata indicator application or maintenance. The reviewer completes the review after checking various types of information of the maintained metadata indicator. The status of the reviewed metadata indicator changes to valid and can be included in the metadata indicator processing task.

[0116] The metadata indicator processing function is the process of incorporating metadata indicator processing rules into metadata indicator processing tasks. Through the execution of tasks, the designed and maintained metadata indicators can be generated. A metadata indicator processing task can include multiple metadata indicators.

[0117] Examples of metadata indicator processing tasks are as follows: Task name: Data standard compliance series calculation tasks; Contains metadata indicators: data item-level data standard compliance, table-level data standard compliance, and application-level data standard compliance.

[0118] Processing logic description: a. Data item level data standard compliance: Variable definition description part: APPNAME=APPLIST.ALL / / This indicator processes and selects the entire application list; SCHNAME=APPNAME.ALL / / This indicator is processed and selected to apply the entire SCHEMA; TABNAME=SCHNAME.ALL / / This indicator selects the full scale under SCHEMA; COLNAME=TABNAME.ALL / / All columns under the processing selection table of this indicator; A1=Basic indicator. Whether the data type is consistent; A2 = basic index. Whether the data length is consistent; A3 = Basic indicators. Whether the data accuracy is consistent; A4=Basic indicators. Code item conformity; A5=Basic indicators. Data item name conformity; P=[50%, 20%, 10%, 15%, 5%] / / Here, the weights used by different basic indicators are defined; B1=composite indicator, data item level data standard compliance: Processing logic part: <LOOP COLNAME> ; B1=(IF(A1,1,0)*P[1]+IF(A2,1,0)*P[2]+IF(A3,1,0)*P[3]+IF(A4,A4,0)*P[4]+IF(A5,A5,0 )*P[5]) / (IF(A1,P[1],0)+IF(A2,P[2],0)+IF(A3,P[3],0)+IF(A4,P[4],0)+IF(A5,P[5],0)); <LOOP END> .

[0119] b. Table-level data standard compliance: Variable definition description part: APPNAME=APPLIST.ALL / / This indicator selects the entire application list; SCHNAME=APPNAME.ALL / / Select and apply the full SCHEMA under this indicator processing; TABNAME=SCHNAME.ALL / / Select the full scale under SCHEMA for this indicator processing; COLP = composite index. Importance of data items; B1=composite index. Data item level data standard compliance; C1 = composite index. Table-level data standard compliance; Processing logic part: <LOOP TABNAME> ; C1C1=IF(B1,B1,0)*COLP++; C1C2=IF(B1,COLP,0)++; C1=C1C1 / C1C2; <LOOP END> .

[0120] c. Application-level data standards compliance: Variable definition description part: APPNAME=APPLIST.ALL / / This indicator processes and selects the entire application list; TABP=composite index. Table importance; C1 = composite index. Table-level data standard compliance; D1 = composite index. Application-level data standard compliance; Processing logic part: <LOOP APPNAME> ; D1D1=IF(C1,C1,0)*TABP++; D1D2=IF(C1,TABP,0)++; D1=D1D1 / D1D2; <LOOP END> .

[0121] The metadata indicator scheduling function is to integrate the maintenance execution cycle, execution time and other information of the metadata indicator processing tasks into the scheduling platform for unified scheduling and execution, and finally generate the metadata indicator processing calculation results. The task scheduling execution results are uniformly queried, displayed and stored.

[0122] The metadata warning rule maintenance function requires the input of warning rule number, warning rule name, warning level, warning cycle, warning environment and other information, and the selection of metadata data indicators that have been maintained and used. After the input is completed, submit it to enter the warning rule review process. In addition, the metadata warning rule maintenance function can be used to modify the rules that have not yet been included in the metadata warning rule processing task and submit them for review again.

[0123] The maintained metadata warning rules are reviewed. After passing the review, the metadata warning rules can be added to the rule processing tasks for subsequent warning scheduling.

[0124] Through the metadata warning rule processing function, the approved metadata warning rules are added to the processing task for subsequent scheduling, execution and monitoring. For rules with the same warning cycle and warning environment, they can be added to the same warning rule processing task in batches.

[0125] The maintenance execution cycle and execution time of the metadata early warning rule processing task are all included in the scheduling platform for unified scheduling and execution, and finally the metadata indicator processing calculation results are generated. The task scheduling execution results are uniformly queried and displayed in the warehouse.

[0126] The warning information generated by the metadata warning rule processing task is color-coded according to the level. Yellow-level warnings are notified by email, and red-level warnings are notified by email and SMS. After the relevant responsible person performs signal processing and maintenance, the metadata warning rule processing task will be rescheduled the next day until the warning signal is confirmed to be cleared.

[0127] The metadata comprehensive analysis module mainly performs independent processing and analysis based on the generated metadata definitions, metadata indicators, metadata indicator processing tasks, metadata warning rules, metadata warning processing tasks and other related information to meet the needs of actual work.

[0128] In some embodiments, the metadata management system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the metadata management system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) metadata management capabilities.

[0129] In this embodiment, the metadata management system can be divided into multiple functional modules according to the functions it performs, such as Figure 4 As shown. The functional modules of the system 400 may include: an attribute generation module 410, a data storage module 420, an index calculation module 430 and an abnormal warning module 440. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0130] An attribute generation module, used to extract feature information of metadata and generate attribute information based on the feature information; A data storage module, configured to assign a key prefix to the metadata based on the attribute information, and save the metadata in a pre-assigned logical interval of the key prefix in the form of a key-value pair; An index calculation module, used to calculate the index value of the attribute information of the metadata in each logical interval respectively using a preset index rule; The abnormal warning module is used to confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate a metadata abnormal warning.

[0131] Optionally, as an embodiment of the present invention, the attribute generation module includes: A feature extraction unit is used to extract feature information from the ledger and metadata directory using an entity naming model, wherein the feature information includes attribution source information, technical information, and resource information; the attribution source information includes attribution application system, business management identity information, and technical management identity information; the technical information includes database information, mode information, table-level information, field-level information, and index information; the resource information includes data resource attribution identity information, data resource status, and attribution development platform information; A first setting unit is used to set the attribution source information as the management attribute of the metadata, set the technical information as the technical attribute of the metadata, and set the resource information as the resource archiving attribute of the metadata; The second setting unit is used to generate standard comparison information according to the technical information and pre-set technical standard parameters, and whether the technical information conforms to the corresponding technical standard parameters, and set the standard comparison information as a data matching attribute of the metadata; the technical standard parameters include a mapping standard name corresponding to a combination of a table name and a field name, a standard data type, a standard data length, a standard data precision and a standard code item.

[0132] Optionally, as an embodiment of the present invention, the data storage module includes: A data classification unit, used to determine the data type of the metadata according to the attribute information; an identification allocation unit, configured to allocate a key prefix to the metadata according to a correspondence between the data type and the key prefix and the determined data type; A configuration storage unit is used to pre-allocate a logical interval for each key prefix, and store the data type, key prefix and logical interval address of the metadata in a configuration file; The encoding storage unit is used to serialize the metadata and store the processed metadata in a corresponding logical interval.

[0133] Figure 5The metadata management method provided for the embodiment of the present application can be applied to the device. It can be understood by those skilled in the art that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device may include more or less components than shown, or combine certain components, or arrange different components. In the embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0134] The device 500 may include: a processor 510, a memory 520 and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0135] The memory 520 may be used to store the execution instructions of the processor 510, and the memory 520 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the device 500 is enabled to perform some or all of the steps in the following method embodiments.

[0136] The processor 510 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 520, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 510 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0137] The communication unit 530 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0138] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0139] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention.

[0140] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0141] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0142] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0144] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person skilled in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, and they shall be within the scope of protection of the present invention.

Claims

1. A metadata management method, characterized in that: include: Extracting feature information of metadata, and generating attribute information based on the feature information; assigning a key prefix to the metadata based on the attribute information, and saving the metadata in a pre-assigned logical interval of the key prefix in the form of a key-value pair; Calculate the index value of the attribute information of the metadata in each logical interval respectively by using the preset index rule; Confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate a metadata abnormality warning.

2. The method according to claim 1, characterized in that: Extracting feature information of metadata and generating attribute information based on the feature information includes: The entity naming model is used to extract characteristic information from the ledger and metadata directory, wherein the characteristic information includes attribution source information, technical information, and resource information; the attribution source information includes attribution application system, business management identity information, and technical management identity information; the technical information includes database information, mode information, table-level information, field-level information, and index information; the resource information includes data resource attribution identity information, data resource status, and attribution development platform information; Set the attribution source information as the management attribute of the metadata, set the technical information as the technical attribute of the metadata, and set the resource information as the resource archiving attribute of the metadata; According to the technical information and pre-set technical standard parameters, standard comparison information is generated according to whether the technical information conforms to the corresponding technical standard parameters, and the standard comparison information is set as a data matching attribute of the metadata; the technical standard parameters include a mapping standard name corresponding to a combination of a table name and a field name, a standard data type, a standard data length, a standard data precision and a standard code item.

3. The method according to claim 1, characterized in that Allocating a key prefix to the metadata based on the attribute information, and saving the metadata in the form of a key-value pair to a pre-allocated logical interval of the key prefix, including: Determine the data type of the metadata according to the attribute information; assigning a key prefix to the metadata according to a correspondence between the data type and the key prefix and the determined data type; Assign a logical interval to each key prefix in advance, and store the data type, key prefix, and logical interval address of the metadata in a configuration file; The metadata is serialized and stored in the corresponding logical interval.

4. The method according to claim 2, characterized in that: The index values ​​of the attribute information of the metadata in each logical interval are calculated using the preset index rules, including: Set the weights of each parameter item of the data benchmarking attribute and calculate the weighted sum to obtain the data item-level standard compliance; Perform weighted summation based on the preset data item weights and item-level standard compliance to obtain table-level standard compliance; A weighted sum is performed according to the preset table weight and the table-level standard compliance to obtain the application-level standard compliance.

5. The method according to claim 4, characterized in that Confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate metadata abnormality warning, including: A threshold combination is set for each logical interval respectively, wherein the threshold combination includes a data item level standard compliance threshold, a table level standard compliance threshold and an application level standard compliance threshold; Compare the data item-level standard compliance, table-level standard compliance and application-level standard compliance within the logical interval with the corresponding thresholds. If they are lower than the corresponding thresholds, a corresponding abnormal warning is generated; The One-Class SVM model is used to analyze the boundary value of each logical interval and determine the abnormal metadata based on the boundary value.

6. A metadata management system, characterized in that: include: An attribute generation module, used to extract feature information of metadata and generate attribute information based on the feature information; A data storage module, configured to assign a key prefix to the metadata based on the attribute information, and save the metadata in a pre-assigned logical interval of the key prefix in the form of a key-value pair; An index calculation module, used to calculate the index value of the attribute information of the metadata in each logical interval respectively using a preset index rule; The abnormal warning module is used to confirm that the indicator value of the logical interval exceeds the set indicator threshold and generate a metadata abnormal warning.

7. The system according to claim 6, characterized in that The attribute generation module comprises: A feature extraction unit is used to extract feature information from the ledger and metadata directory using an entity naming model, wherein the feature information includes attribution source information, technical information, and resource information; the attribution source information includes attribution application system, business management identity information, and technical management identity information; the technical information includes database information, mode information, table-level information, field-level information, and index information; the resource information includes data resource attribution identity information, data resource status, and attribution development platform information; A first setting unit is used to set the attribution source information as the management attribute of the metadata, set the technical information as the technical attribute of the metadata, and set the resource information as the resource archiving attribute of the metadata; The second setting unit is used to generate standard comparison information according to the technical information and pre-set technical standard parameters, and whether the technical information conforms to the corresponding technical standard parameters, and set the standard comparison information as a data matching attribute of the metadata; the technical standard parameters include a mapping standard name corresponding to a combination of a table name and a field name, a standard data type, a standard data length, a standard data precision and a standard code item.

8. The system according to claim 6, characterized in that The data storage module comprises: A data classification unit, used to determine the data type of the metadata according to the attribute information; an identification allocation unit, configured to allocate a key prefix to the metadata according to a correspondence between the data type and the key prefix and the determined data type; A configuration storage unit is used to pre-allocate a logical interval for each key prefix, and store the data type, key prefix and logical interval address of the metadata in a configuration file; The encoding storage unit is used to serialize the metadata and store the processed metadata in a corresponding logical interval.

9. A metadata management device, characterized in that: include: A memory for storing a metadata management program; A processor, configured to implement the steps of the metadata management method according to any one of claims 1 to 5 when executing the metadata management program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a metadata management program, and when the metadata management program is executed by a processor, the steps of the metadata management method according to any one of claims 1 to 5 are implemented.

Images