Metadata-driven data resource catalog management method

Abstract

The application provides a metadata-driven data resource directory management method, and relates to the technical field of data management, comprising the following steps: using a data warehouse to regularly scan a data source to obtain metadata resources, managing the metadata resources; classifying and labeling the metadata resources based on a set classification rule; binding the metadata resources with permissions or roles to form a target data map; regularly updating a data resource directory and providing a data service interface and a search function for each data resource. Through the above steps, the metadata resources obtained by regularly scanning the data source using the data warehouse are managed; the metadata resources are classified and labeled based on the set classification rule; the metadata resources are bound with permissions or roles to form the target data map; the data resource directory is regularly updated, and the data service interface and the search function are provided for each data resource, so that the utilization efficiency of the data resources, the accuracy of data resource management, the security of data and the support capability of business decision-making can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to a metadata-driven data resource directory management method. BACKGROUND

[0002] In recent years, with the continuous deepening of enterprise informatization construction, the management and utilization of data resources have become increasingly important. Traditional data resource management methods often have problems such as scattered data resources, difficulty in unified management and discovery, resulting in low utilization of data resources and inability to meet the growing data needs of enterprises. Therefore, how to efficiently manage and utilize data resources has become one of the current research focuses.

[0003] Therefore, the present application provides a metadata-driven data resource directory management method. SUMMARY

[0004] The present application provides a metadata-driven data resource directory management method, which manages the metadata resources obtained by periodically scanning the data sources using a data warehouse; classifies and labels the metadata resources based on a set classification rule; binds the metadata resources with permissions or roles to form a target data map; regularly updates the data resource directory and provides data service interfaces and search functions for each data resource, which can effectively improve the utilization efficiency of data resources, the accuracy of data resource governance, the security of data, and the support capability of business decision-making.

[0005] The present application provides a metadata-driven data resource directory management method, which includes:

[0006] Step 1: Use a target data warehouse to periodically scan data sources to obtain metadata resources, and manage the metadata resources;

[0007] Step 2: Classify the metadata resources based on a set classification rule and label the corresponding functional tags;

[0008] Step 3: Bind the metadata resources with permissions or roles to form a target data map;

[0009] Step 4: Regularly update the data resource directory and provide data service interfaces and search functions for each data resource.

[0010] Preferably, based on the set classification rule, the metadata resources are classified and labeled with corresponding functional tags, which includes:

[0011] Based on the set classification rule, the metadata resources are classified into three levels according to business functions to obtain business-classification data;

[0012] The business-classification data is classified according to the data type, and the number-classification data is obtained;

[0013] The business-classification data is classified according to the data type, and the number-classification data is obtained;

[0014] The business-classification data is classified according to the data type, and the number-classification data is obtained;

[0015] The business-classification data is classified according to the data type, and the number-classification data is obtained;

[0016] The business-classification data is classified according to the data type, and the number-classification data is obtained;

[0017] Preferably, the metadata resources are bound to permissions or roles to form a target data map, including:

[0018] The metadata resources and the corresponding level function tags are used as matching conditions to extract the first role and the first permission group from the set role-permission database;

[0019] In the metadata management system, the first role and the first permission group are bound to the current metadata resource;

[0020] According to the preset map layout requirements, a set drawing tool is used to draw an initial data map;

[0021] The metadata resources and the corresponding function tags, the first role and the first permission group are input into the set drawing tool;

[0022] The set drawing tool establishes the relationship between the metadata resources, the function tags, the first role and the first permission group, adjusts the layout of the initial data map, and forms a target data map.

[0023] Preferably, the data resource directory is updated regularly, and a data service interface and a search function are provided for each data resource, including:

[0024] The target data resource directory system is established by using the data resources and the corresponding metadata resources;

[0025] Based on the monitoring characteristics of the data resources, different update mechanisms are used to regularly update the metadata resources in the target data resource directory system;

[0026] The metadata resources in the target data resource directory are imported into a set search engine, metadata indexes are established, a plurality of set search modes are provided, and the set search engine is integrated into the target data resource directory system.

[0027] Based on user needs and the characteristics of data resources, the functions and parameters of the data service interface are defined, and then the data service interface is developed.

[0028] The data service interface shall be maintained regularly, and corresponding remedial measures shall be taken to fix any vulnerabilities found.

[0029] Use pre-deployed monitoring tools to monitor the usage of data service interfaces in real time and obtain interface performance data;

[0030] The interface performance data is input into a pre-established anomaly detection model. After anomaly detection and labeling of the interface performance data, the anomaly detection result is output.

[0031] If the anomaly detection results contain interface performance data marked as abnormal, it is determined that the current data service interface has an interface anomaly. The interface performance data marked as abnormal is used as a matching condition to match the corresponding anomaly maintenance strategy for maintenance.

[0032] Preferably, the set search method includes four search methods: keyword search, category search, attribute search, and search by combination of multiple search conditions.

[0033] Preferably, based on the monitoring characteristics of data resources, different update mechanisms are used to periodically update the metadata resources in the target data resource catalog system, including:

[0034] If the monitoring characteristics of the data resource support automated monitoring, then the current data resource is regarded as the first data resource;

[0035] Using a set monitoring tool, periodic change detection is performed on the first data resource in the target data resource catalog system according to a preset dynamic update cycle, and the change detection results are obtained.

[0036] Based on the change detection results, if the first data resource has changed, the corresponding metadata resource of the current first data resource in the target data resource catalog system will be automatically updated according to the change detection results.

[0037] If the monitoring characteristics of the data resource do not support automated monitoring, then the current data resource will be regarded as the second data resource.

[0038] For the second data resource in the target data resource catalog system, a target update request is periodically submitted according to a preset dynamic update cycle using a set manual update strategy;

[0039] The target update request is reviewed, and when the review is successful, the corresponding metadata resource of the current second data resource in the target data resource catalog system is updated.

[0040] Preferably, it also includes:

[0041] Obtain the historical change data of the current data resource in a preset time period from the data resource change library, and input the change prediction model established in advance to obtain the predicted change data of the current data resource;

[0042] Based on the predicted change time point of the current data resource in the predicted change data as the independent variable and the predicted metadata change amount as the dependent variable, an estimated resource change sequence graph is established;

[0043] Extract statistical analysis indicators from the estimated resource change sequence graph, and perform change trend analysis to obtain an estimated change trend coefficient of the current data resource;

[0044] Based on the set business evaluation indicators, the business demand of the current data resource is evaluated to obtain a demand evaluation indicator value;

[0045] The obtained demand evaluation indicator value is combined with the estimated change trend coefficient for analysis, and a resource update demand coefficient is calculated;

[0046] The calculation formula of the resource update demand coefficient is as follows:

[0047] In the formula, represents the resource update demand coefficient of the current data resource; represents the metadata change amount of the current data resource at the current time; represents the predicted metadata change amount of the jth predicted change time point of the current data resource, where j=1, 2, 3, , n; represents the time interval between the current time and the jth predicted change time point; represents the estimated change trend coefficient of the current data resource; represents the influence weight of the predicted change trend of the metadata on changing the data resource update period; represents the influence weight of the business demand on changing the data resource update period; represents the ith set business evaluation indicator, where i=1, 2; represents the contribution weight of the ith set business evaluation indicator to evaluating the business demand of the current data resource; e represents a constant, and the value is 2.7;

[0048] When the resource update demand coefficient belongs to the set demand threshold range, it is determined that the preset dynamic update period of the current data resource does not need to be updated;

[0049] When the resource update demand coefficient does not fall within the set demand threshold range, it is determined that the current data resource needs to be updated within the preset dynamic update cycle.

[0050] By analyzing the predicted changes in current data resources and the historical dynamic update cycle, the real-time dynamic cycle is obtained.

[0051] The update is achieved by replacing the current preset dynamic update cycle with the real-time dynamic cycle.

[0052] Preferably, the real-time dynamic cycle is obtained by analyzing the predicted changes in current data resources and the historical dynamic update cycle, including:

[0053] Extract the historical dynamic update cycle of the current data resources within a preset time period from the data resource change record library, and output it as the first analysis cycle;

[0054] The historical generation time point of the first analysis cycle is output as the first update time.

[0055] If there is no first analysis period, the resource update demand coefficient of the current data resources will be combined with the predicted change data to calculate the first dynamic analysis period.

[0056] The formula for calculating the first dynamic cycle is as follows:

[0057] In the formula, This indicates the first dynamic analysis cycle; this indicates the current preset dynamic update cycle. This is expressed as the resource update demand coefficient; This indicates setting an upper limit for the demand threshold; This represents the time interval between the predicted time point corresponding to the minimum change in the amount of metadata resources in the predicted change data and the current time. This represents the time interval between the predicted time point corresponding to the maximum change in metadata resource quantity in the predicted change data and the current time; e is a constant with a value of 2.7. This represents the weight of the impact of the current data resource update demand on the calculation of the first dynamic analysis cycle; This represents the weight of the impact of the current data resource change prediction on the calculation of the first dynamic analysis cycle; This represents the maximum change in the amount of metadata resources in the predicted change data; Represented as the minimum change in metadata resource quantity in the predicted change data.

[0058] The first analysis dynamic cycle is output as the real-time dynamic cycle.

[0059] If there is a single first analysis period, a second analysis dynamic period is calculated according to a historical time interval between the first update time of the current first analysis period and the current time and a historical resource change amount;

[0060] The calculation formula of the second analysis dynamic period is as follows:

[0061] In the formula, the second analysis dynamic period is represented as T2; The second analysis dynamic period is represented as T2; The influence weight of the resource demand and the resource change prediction on the calculation of the second analysis dynamic period is represented as W2; The influence weight of the historical update situation of the preset dynamic update period of the current data resource on the calculation of the second analysis dynamic period is represented as W1; The historical time interval between the first update time of the current first analysis period and the current time is represented as T; The predicted time period is represented as Tp; The resource absolute difference between the metadata resource amount at the predicted last time and the metadata resource amount at the current time is represented as D; The resource absolute difference between the data resource amount at the current first update time and the data resource amount at the current time is represented as D1;

[0062] The second analysis dynamic period is output as a real-time dynamic period;

[0063] If there are multiple first analysis periods, a third analysis dynamic period is calculated according to the historical update interval between the corresponding first update times of all the first analysis periods and the historical resource change amount;

[0064] The calculation formula of the third analysis dynamic period is as follows:

[0065] In the formula, the third analysis dynamic period is represented as T3; The third analysis dynamic period is represented as T3; The cth first analysis period is represented as Tc, c = 1, 2, n; n represents the total number of the first analysis periods; The resource absolute difference between the metadata resource amount at the corresponding first update time of the cth first analysis period and the metadata resource amount at the current time is represented as Dc;

[0066] The third analysis dynamic period is output as a real-time dynamic period.

[0067] Compared with the prior art, the application has the following beneficial effects:

[0068] The metadata resources obtained by regularly scanning the data source by using the data warehouse are managed, the metadata resources are classified and labeled based on the set classification rules, the metadata resources are bound with permissions or roles to form a target data map, the data resource directory is regularly updated, and the data service interface and search function are provided for each data resource, so that the utilization efficiency of the data resources and the accuracy of the data resource management are effectively improved, the data security is enhanced, and the support capability of the business decision is strengthened.

[0069] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims thereof.

[0070] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application, and explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings:

[0072] Figure 1 The flow chart of a metadata-driven data resource directory management method in the embodiments of the present application. DETAILED DESCRIPTION

[0073] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to explain and illustrate the present application, and do not constitute a limitation of the present application.

[0074] The embodiments of the present application provide a metadata-driven data resource directory management method, as shown in Figure 1 , comprising:

[0075] Step 1: using a target data warehouse to regularly scan a data source to obtain metadata resources, and managing the metadata resources;

[0076] Step 2: classifying the metadata resources and labeling corresponding function tags based on set classification rules;

[0077] Step 3: binding the metadata resources with permissions or roles to form a target data map;

[0078] Step 4: regularly updating the data resource directory, and providing a data service interface and search function for each data resource.

[0079] In this embodiment, the target data warehouse is a large, centralized storage system for storing and managing data extracted, transformed and loaded (ETL) from various data sources; the data sources are predetermined, such as databases, file systems, data lakes; the metadata resource refers to the relevant information of the data resource, such as the structure, content, source, format, quality, access permission, etc. of the data; the function label includes three kinds of business function label, number class function label and level function label; the target data map is a visual tool or representation that shows the relationship between the metadata resource, the function label, the first role and the first permission group; the data resource directory is a list of data resources within an organization or institution, which is composed of data resources and corresponding metadata resources.

[0080] In this embodiment, the target data warehouse is used to periodically scan all data sources to obtain metadata resources, and the metadata resources are managed, including:

[0081] The target data warehouse is used to periodically scan all data sources according to a preset time period to obtain all data tables, fields and metadata;

[0082] The obtained metadata is labeled as metadata resources;

[0083] The metadata resources are imported into a set metadata management tool for centralized storage and maintenance;

[0084] The preset time period is predetermined, such as every week; the set metadata management tool is predetermined, which is a tool for centralized storage and maintenance of metadata resources.

[0085] The beneficial effects of the above technical solution are: the metadata resources obtained by periodically scanning the data sources using the data warehouse are managed; the metadata resources are classified and labeled based on the set classification rules; the metadata resources are bound to permissions or roles to form a target data map; the data resource directory is updated regularly, and a data service interface and search function are provided for each data resource, which can effectively improve the utilization efficiency of data resources, the accuracy of data resource management, the security of data, and the support capability of business decision-making.

[0086] The embodiment of the application provides a metadata-driven data resource directory management method, which classifies metadata resources and labels corresponding function labels based on a set classification rule, including:

[0087] According to the set classification rule, the metadata resources are classified into three levels according to business functions to obtain business-classification data;

[0088] The business-classification data is labeled with corresponding business function labels, and the business function labels are added in the metadata management system;

[0089] According to the data type, the service-classified data is secondarily classified to obtain the number-classified data;

[0090] The corresponding data type label is created for the number-classified data, and the number-class function label is added in the metadata management system;

[0091] According to the data security level, the number-classified data is primarily classified to obtain the security-classified data;

[0092] The corresponding data security level label is created for the security-classified data, and the level function label is added in the metadata management system.

[0093] In this embodiment, the function label includes the service function label, the number-class function label and the level function label; the service-classified data is obtained by classifying the metadata resources according to the service function, wherein the service function includes sales, finance, human resources and the like; the number-classified data is obtained by classifying the service-classified data according to the data type, wherein the data type includes structured data and unstructured data; and the security-classified data is obtained by classifying the number-classified data according to the data security level, wherein the data security level includes the general level, the important level and the core level.

[0094] The above technical scheme has the beneficial effects that: by classifying the metadata resources based on the set classification rules and labeling the corresponding function labels, data basis can be provided for establishing the data map, and through the classification and labeling, the metadata resources can be more orderly organized and stored, so that the searching and retrieving become more rapid and accurate, and the resource management efficiency is improved.

[0095] The metadata-driven data resource directory management method provided by the embodiment of the application binds the metadata resources to the permissions or roles to form a target data map, and includes the following steps:

[0096] Taking the metadata resources and the corresponding level function labels as the matching conditions, the first role and the first permission group are extracted from the set role-permission database;

[0097] In the metadata management system, the first role and the first permission group are bound to the current metadata resources;

[0098] According to the preset map layout requirement, the set drawing tool is used to draw an initial data map;

[0099] The metadata resources and the corresponding function labels, the first role and the first permission group are input into the set drawing tool;

[0100] The setting drawing tool establishes the relationship between the metadata resource, the function label, the first role and the first permission group, adjusts the layout of the initial data map, and forms a target data map.

[0101] In this embodiment, the setting role-permission database is a pre-established database that stores different roles and the permissions that these roles have, such as reading, writing, and deleting permissions of resources; the first role refers to one or more roles that are extracted from the setting role-permission database and match the current metadata resource and the corresponding level function label, and these roles represent users or user groups that can access or operate the current metadata resource; the first permission group refers to a set of permissions that are extracted from the setting role-permission database and match the current metadata resource and the corresponding level function label corresponding to the first role, such as reading data and writing data; the metadata management system is a system for storing, managing and maintaining metadata of data resources, which can define, classify, label and manage metadata of various data resources to better understand and utilize these data resources; the preset map layout requirement refers to the map layout requirement set according to the specific needs of an organization or institution before drawing an initial data map, such as the arrangement of data resources, color coding, and label display mode; the initial data map refers to an initial map drawn by using the setting drawing tool according to the preset map layout requirement, wherein the setting drawing tool is a tool pre-determined for drawing a map; the target data map refers to a map drawn by using the setting drawing tool, which displays metadata resources and their related information (such as function labels, roles, permissions, etc.).

[0102] The above technical solution has the beneficial effects that: by binding the metadata resources with permissions or roles, a target data map is formed, which can effectively prevent data leakage and misuse, simplify data management operations, improve data quality, and promote data sharing and collaboration.

[0103] The embodiment of the present application provides a metadata-driven data resource directory management method, which periodically updates the data resource directory and provides data service interfaces and search functions for each data resource, comprising:

[0104] A target data resource directory system is established by using data resources and corresponding metadata resources;

[0105] Based on the monitoring characteristics of the data resources, different update mechanisms are used to periodically update the metadata resources in the target data resource directory system;

[0106] The metadata resources in the target data resource directory are imported into a setting search engine, metadata indexes are established, and multiple setting search methods are provided, and then the setting search engine is integrated into the target data resource directory system;

[0107] Based on the user requirements and the characteristics of the data resources, define the functions and parameters of the data service interface, and develop the data service interface;

[0108] Periodically maintain the data service interface and take appropriate repair measures to repair when there are vulnerabilities;

[0109] Use the pre-deployed setting monitoring tool to monitor the usage of the data service interface in real time to obtain interface performance data;

[0110] Input the interface performance data into the pre-established anomaly detection model, perform anomaly judgment and labeling on the interface performance data, and output the anomaly detection results;

[0111] If there is interface performance data labeled as abnormal in the anomaly detection results, it is determined that the current data service interface has interface anomalies, and the interface performance data labeled as abnormal is used as a matching condition to match the corresponding anomaly maintenance strategy for maintenance.

[0112] In this embodiment, the target data resource directory system refers to a system for organizing and storing data resources, which constructs a structured directory by using data resources and their corresponding metadata, so that users can more easily find, access and manage resources; the monitoring characteristics refer to support for automated monitoring and not support for automated monitoring; the update mechanism refers to the rules for periodically updating metadata resources in the data resource directory system, and different monitoring characteristics use different update mechanisms.

[0113] In this embodiment, the setting search engine is pre-determined, such as Elasticsearch, Solr; the setting search method includes keyword search, classification search, attribute search, and multi-search condition combination search; the user requirements refer to data content, format, quality, and access requirements; the characteristics of the data resources refer to data type, data format, data source, data quality, and data access permission; the data service interface refers to an interface provided for the target data resource directory system, which is used to interact with external systems or applications, such as an API interface; defining the functions and parameters of the data service interface refers to specifying the service types, service contents, and required parameters of the data service interface, etc. information; the repair measures refer to the repair and improvement measures taken when vulnerabilities or problems are found during periodic maintenance of the data service interface, such as updating interface code, repairing security vulnerabilities, optimizing interface performance, etc.

[0114] In the embodiment, the interface performance data refers to data generated in the use process of the data service interface, reflecting the interface performance and stability, such as response time and throughput; the set monitoring tool refers to a software tool for monitoring the running of the data service interface in real time; the anomaly detection model refers to a model for identifying and analyzing the performance anomaly problems of the data service interface, which is obtained by training a neural network based on historical data or predefined rules; the anomaly detection result refers to the labeling result obtained after the interface performance data is abnormally judged; the anomaly maintenance strategy refers to an anomaly handling strategy matched from the set anomaly handling table by taking the interface performance data labeled as abnormal as a matching condition, wherein the anomaly handling strategy refers to a predefined maintenance step or maintenance operation, which is used to restore the normal operation of the system when the interface anomaly is detected, such as restarting the service; and the set anomaly handling table is composed of interface performance data and corresponding anomaly handling strategies.

[0115] The beneficial effects of the above technical solution are: by periodically updating the data resource directory and providing data service interfaces and search functions for each data resource, the timeliness of data can be ensured and data sharing and cooperation can be promoted.

[0116] The embodiment of the application provides a metadata-driven data resource directory management method, based on the monitoring characteristics of data resources, different update mechanisms are used to periodically update the metadata resources in the target data resource directory system, comprising:

[0117] If the monitoring characteristics of the data resources support automatic monitoring, the current data resources are regarded as first data resources;

[0118] The set monitoring tool is used to periodically change the first data resources in the target data resource directory system according to the preset dynamic update period to obtain a change detection result;

[0119] According to the change detection result, if the first data resources change, the corresponding metadata resources of the current first data resources in the target data resource directory system are automatically updated according to the change detection result;

[0120] If the monitoring characteristics of the data resources do not support automatic monitoring, the current data resources are regarded as second data resources;

[0121] The second data resources in the target data resource directory system are periodically submitted with a target update request according to a preset dynamic update period by using a set manual update strategy;

[0122] The target update request is audited, and when the audit is successful, the corresponding metadata resources of the current second data resources in the target data resource directory system are updated.

[0123] In the embodiment, the first data resource refers to a data resource with a monitoring characteristic of supporting automatic monitoring; the preset dynamic update period refers to a set frequency of data resource change detection or update request submission; the set monitoring tool refers to a tool pre-configured or specified for periodic change detection of the data resource, which can be a software program, a script or other automation means, and is used for monitoring the data resource in the target data resource directory system according to the preset dynamic update period to detect changes thereof; the change detection result refers to a result obtained after periodic change detection of the first data resource in the target data resource directory system, and is used for reflecting changes of the data resource in a period of time, such as addition, deletion, modification and the like; the second data resource refers to a data resource with a monitoring characteristic of not supporting automatic monitoring; and the set manual update strategy refers to a pre-set update strategy, which is composed of a request submission mode, a data identifier, an update requester and an update content description and the like. The target update request is composed of an update time, an update content and an update reason.

[0124] The beneficial effects of the above technical solution are: by using different update mechanisms to periodically update the metadata resource in the target data resource directory system, the scalability and flexibility of the system can be enhanced, and the management and utilization efficiency of the data resource can be improved.

[0125] The embodiment of the application provides a metadata-driven data resource directory management method, which further comprises:

[0126] The historical change data of the current data resource in a preset time period is obtained from the data resource change record library and input into a pre-established change prediction model to obtain predicted change data of the current data resource;

[0127] Based on the predicted change time point of the current data resource in the predicted change data as the independent variable and the predicted metadata change amount as the dependent variable, an estimated resource change sequence graph is established;

[0128] Statistical analysis indicators are extracted from the estimated resource change sequence graph, and an estimated change trend coefficient of the current data resource is obtained through change trend analysis;

[0129] Based on the set business evaluation index, the business demand of the current data resource is evaluated to obtain a demand evaluation index value;

[0130] The obtained demand evaluation index value is combined with the estimated change trend coefficient for analysis, and a resource update demand coefficient is calculated;

[0131] The calculation formula of the resource update demand coefficient is as follows:

[0132] In the formula, the resource update demand coefficient of the current data resource is represented by ​ represents the metadata change amount of the current data resource at the current time point; represents the predicted metadata change amount of the current data resource at the jth predicted change time point, wherein j = 1, 2, 3, , n; represents the time interval between the current time point and the jth predicted change time point; represents the estimated change trend coefficient of the current data resource; represents the influence weight of the predicted change trend of the metadata on changing the data resource update period; represents the influence weight of the business demand on changing the data resource update period; represents the ith set business evaluation index, wherein i = 1, 2; represents the contribution weight of the ith set business evaluation index to evaluating the business demand of the current data resource; e represents a constant, and the value is 2.7;

[0133] When the resource update demand coefficient belongs to the set demand threshold range, it is determined that the preset dynamic update period of the current data resource does not need to be updated;

[0134] When the resource update demand coefficient does not belong to the set demand threshold range, it is determined that the preset dynamic update period of the current data resource needs to be updated;

[0135] By analyzing the predicted change data of the current data resource and the historical dynamic update period, a real-time dynamic period is obtained;

[0136] The real-time dynamic period is used to replace the current preset dynamic update period, so that the update is realized.

[0137] In this embodiment, the data resource change record library refers to a database for storing data resource change records, and the change records include information such as change time, type, content and the like; the preset time period is determined in advance, such as 6 months; the historical change data includes historical change time points, historical change frequency and historical change amount of the data resource and the like; the change prediction model is a prediction model obtained by training a neural network based on historical data resource change records and characteristics of the data resource, and is used to predict the change of the data resource in a future period of time.

[0138] In the embodiment, the prediction change data includes a prediction change time point, a prediction change frequency, a prediction change amount, etc.; the estimation resource change sequence diagram is established by taking the prediction change time point of the data resource as the independent variable and the prediction metadata change amount as the dependent variable; the statistical analysis index is a quantitative index for measuring and analyzing the data resource change, such as the average value and the standard deviation of the change amount; and the estimation change trend coefficient is used to represent the prediction resource change degree of the current data resource, and is obtained by weighted average of the index values of the extracted statistical analysis indexes, wherein the weight given to the index value is obtained by solving a matrix constructed by pairwise comparison and relative importance scoring using the analytic hierarchy process.

[0139] In the embodiment, the set business evaluation index refers to the data resource update timeliness and data resource stability; the demand evaluation index value refers to the index value obtained by evaluating the business demand of the current data resource based on the set business evaluation index; the resource update demand coefficient is used to represent the adjustment urgency of the preset dynamic update period of the current data resource; and the set demand threshold range is determined in advance and consists of a set demand threshold upper limit and a set threshold demand lower limit, which is generally ; the historical dynamic update period refers to the preset dynamic update period adopted by the data resource in the past preset time period; and the real-time dynamic period is obtained by analyzing the prediction change data of the current data resource and the historical dynamic update period, and is used to replace the current preset dynamic update period (when it is determined that the preset dynamic update period of the current data resource needs to be updated).

[0140] The beneficial effects of the above technical solution are: by analyzing the update change trend of the data resource and the business demand, the preset dynamic update period is updated, which can effectively avoid data lag caused by too long update period or system burden and resource waste caused by too short update period, thereby improving resource utilization efficiency and reducing operation cost.

[0141] The embodiment of the application provides a metadata-driven data resource directory management method, which obtains a real-time dynamic period by analyzing prediction change data of a current data resource and a historical dynamic update period, and comprises the following steps:

[0142] Extracting, from a data resource change record library, a historical dynamic update period of the current data resource in a preset time period and outputting the historical dynamic update period as a first analysis period;

[0143] Outputting a historical generation time point of the first analysis period as a first update time point;

[0144] If the first analysis period does not exist, combining a resource update demand coefficient of the current data resource with the prediction change data to analyze and calculate a first analysis dynamic period;

[0145] The calculation formula of the first analysis dynamic period is as follows:

[0146] In the formula, is the first analysis dynamic period; and is the current preset dynamic update period; is the resource update demand coefficient; is the set demand threshold upper limit; is the time interval between the corresponding prediction time point of the minimum value of the metadata resource quantity change in the prediction change data and the current time; is the time interval between the corresponding prediction time point of the maximum value of the metadata resource quantity change in the prediction change data and the current time; e is a constant, and the value is 2.7; is the influence weight of the resource update demand degree of the current data resource on the calculation of the first analysis dynamic period; is the influence weight of the resource change prediction of the current data resource on the calculation of the first analysis dynamic period; is the maximum value of the metadata resource quantity change in the prediction change data; is the minimum value of the metadata resource quantity change in the prediction change data

[0147] The first analysis dynamic period is output as a real-time dynamic period;

[0148] If there is a single first analysis period, a second analysis dynamic period is calculated according to the historical time interval between the first update time of the current first analysis period and the current time and the historical resource change quantity;

[0149] The calculation formula of the second analysis dynamic period is as follows:

[0150] In the formula, is the second analysis dynamic period; is the influence weight of the resource demand and the resource change prediction on the calculation of the second analysis dynamic period; is the influence weight of the historical update situation of the preset dynamic update period of the current data resource on the calculation of the second analysis dynamic period; is the historical time interval between the first update time of the current first analysis period and the current time; is the prediction time period; is the resource absolute difference between the metadata resource quantity at the prediction last time and the metadata resource quantity at the current time; is the resource absolute difference between the data resource quantity at the current first update time and the data resource quantity at the current time;

[0151] output the second analysis dynamic period as a real-time dynamic period;

[0152] If there are multiple first analysis periods, a third analysis dynamic period is calculated according to the historical update intervals between the corresponding first update time of all first analysis periods and the historical resource change amount;

[0153] The calculation formula of the third analysis dynamic period is as follows:

[0154] In the formula, the first analysis dynamic period is represented as the third analysis dynamic period. In the formula, the first analysis dynamic period is represented as the third analysis dynamic period. In the formula, the first analysis dynamic period is represented as the third analysis dynamic period. In the formula, the first analysis dynamic period is represented as the third analysis dynamic period. In the formula, the first analysis dynamic period is represented as the third analysis dynamic period.

[0155] The third analysis dynamic period is output as a real-time dynamic period.

[0156] In this embodiment, the first analysis period refers to the historical dynamic update period of the current data resource in a preset time period extracted from the data resource change record library; the historical generation time point refers to the starting time point at which the first analysis period (i.e. the historical dynamic update period) is applied; the first update time refers to the historical generation time point of the first analysis period; the first analysis dynamic period is an update period of the data resource calculated by combining the resource update demand coefficient of the current data resource with the predicted change data (when there is no first analysis period); the second analysis dynamic period is an update period of the data resource calculated based on the analysis of the historical time interval between the first update time of the current first analysis period and the current time and the historical resource change amount (when there is a single first analysis period); and the third analysis dynamic period is an update period of the data resource calculated based on the analysis of the historical update intervals between the corresponding first update time of all first analysis periods and the historical resource change amount (when there are multiple first analysis periods).

[0157] The above technical solution has the beneficial effect that by analyzing the predicted change data of the current data resource and the historical dynamic update period, the real-time dynamic period is obtained to replace the current preset dynamic update period, which can update the preset dynamic update period, thereby effectively avoiding data lag caused by an excessively long update period or system burden and resource waste caused by an excessively short update period, so as to improve resource utilization efficiency.

[0158] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A metadata-driven data resource catalog management method, characterized in that, include: Step 1: Use the target data warehouse to periodically scan the data source to obtain metadata resources and manage these metadata resources; Step 2: Based on the established classification rules, classify the metadata resources and label them with corresponding functional tags; Step 3: Bind permissions or roles to metadata resources to create a target data map; Step 4: Regularly update the data resource catalog and provide data service interfaces and search functions for each data resource; The data resource catalog management method further includes: The historical change data of the current data resources within a preset time period is obtained from the data resource change record library and input into the pre-established change prediction model to obtain the predicted change data of the current data resources. Based on the predicted change time point of the current data resources in the predicted change data as the independent variable and the predicted change amount of metadata as the dependent variable, a sequence diagram of predicted resource change is established. Statistical analysis indicators are extracted from the estimated resource change sequence map, and trend analysis is performed to obtain the estimated change trend coefficient of the current data resources. The business needs of the current data resources are evaluated based on the set business evaluation indicators, and the values ​​of the demand evaluation indicators are obtained. The obtained demand assessment index values ​​are combined with the predicted change trend coefficient for analysis to calculate the resource renewal demand coefficient. The formula for calculating the resource renewal demand coefficient is as follows: In the formula, This represents the resource update demand coefficient for the current data resources; This represents the amount of metadata change of the current data resource at the current moment; This represents the predicted metadata change at the j-th predicted change time point of the current data resource, where j = 1, 2, 3, ..., n; It is represented as the time interval between the current time and the j-th predicted change time point; It is represented as the estimated trend coefficient of change in current data resources; This is represented as the weight of the impact of the predicted change trend of metadata on the change of data resource update cycle; This is represented by the weight of the impact of business requirements on changing the data resource update cycle; Let i be the i-th set business evaluation indicator, where i = 1, 2; This represents the contribution weight of the i-th set business evaluation indicator to the business needs for evaluating current data resources; e is a constant with a value of 2.

7. When the resource update demand coefficient falls within the set demand threshold range, it is determined that the current data resource does not need to be updated during the preset dynamic update cycle. When the resource update demand coefficient does not fall within the set demand threshold range, it is determined that the current data resource needs to be updated within the preset dynamic update cycle. By analyzing the predicted changes in current data resources and the historical dynamic update cycle, the real-time dynamic cycle is obtained. The current preset dynamic update cycle is replaced by the real-time dynamic cycle to achieve the update.

2. The metadata-driven data resource catalog management method according to claim 1, characterized in that, Based on the established classification rules, metadata resources are categorized and labeled with corresponding functional tags, including: Based on the established classification rules, metadata resources are classified into three levels according to business functions to obtain business-classification data. Create corresponding business function tags for the business-classification data, and add the corresponding business function tags in the metadata management system; The business-category data is further classified into two categories based on data type to obtain data-category data. Create corresponding data type tags for the data categories and add corresponding data category function tags in the metadata management system; Based on the data security level, the data is classified into primary categories to obtain secure-classified data. Create corresponding data security level labels for the security-classified data, and add corresponding level function labels in the metadata management system.

3. The metadata-driven data resource catalog management method according to claim 1, characterized in that, To create a target data map, permissions or roles are bound to metadata resources, including: Using metadata resources and corresponding level function tags as matching conditions, the first role and the first permission group are extracted from the role-permission database. In the metadata management system, the first role and the first permission group are bound to the current metadata resource; Based on the preset map layout requirements, the initial data map is drawn using the designated drawing tools. Input the metadata resources, along with the corresponding function tags, first role, and first permission group, into the drawing tool; The setting drawing tool establishes the relationship between metadata resources, function tags, first role and first permission group, adjusts the layout of the initial data map, and forms the target data map.

4. The metadata-driven data resource catalog management method according to claim 1, characterized in that, The data resource catalog is updated regularly, and data service interfaces and search functions are provided for each data resource, including: Establish a target data resource catalog system using data resources and corresponding metadata resources; Based on the monitoring characteristics of data resources, different update mechanisms are used to periodically update the metadata resources in the target data resource catalog system; Import the metadata resources in the target data resource directory into the set search engine, establish a metadata index, provide multiple set search methods, and then integrate the set search engine into the target data resource directory system; Based on user needs and the characteristics of data resources, the functions and parameters of the data service interface are defined, and then the data service interface is developed. The data service interface shall be maintained regularly, and corresponding remedial measures shall be taken to fix any vulnerabilities found. Use pre-deployed monitoring tools to monitor the usage of data service interfaces in real time and obtain interface performance data; The interface performance data is input into a pre-established anomaly detection model. After anomaly detection and labeling of the interface performance data, the anomaly detection result is output. If the anomaly detection results contain interface performance data marked as abnormal, it is determined that the current data service interface has an interface anomaly. The interface performance data marked as abnormal is used as a matching condition to match the corresponding anomaly maintenance strategy for maintenance.

5. The metadata-driven data resource catalog management method according to claim 4, characterized in that, The search settings include four types: keyword search, category search, attribute search, and search using a combination of multiple search criteria.

6. The metadata-driven data resource catalog management method according to claim 4, characterized in that, Based on the monitoring characteristics of data resources, different update mechanisms are used to periodically update the metadata resources in the target data resource catalog system, including: If the monitoring characteristics of the data resource support automated monitoring, then the current data resource is regarded as the first data resource; Using a set monitoring tool, periodic change detection is performed on the first data resource in the target data resource catalog system according to a preset dynamic update cycle, and the change detection results are obtained. Based on the change detection results, if the first data resource has changed, the corresponding metadata resource of the current first data resource in the target data resource catalog system will be automatically updated according to the change detection results. If the monitoring characteristics of the data resource do not support automated monitoring, then the current data resource will be regarded as the second data resource. For the second data resource in the target data resource catalog system, a target update request is periodically submitted according to a preset dynamic update cycle using a set manual update strategy; The target update request is reviewed, and when the review is successful, the corresponding metadata resource of the current second data resource in the target data resource catalog system is updated.

7. The metadata-driven data resource catalog management method according to claim 6, characterized in that, By analyzing the predicted changes in current data resources and the historical dynamic update cycles, the real-time dynamic cycle is obtained, including: Extract the historical dynamic update cycle of the current data resources within a preset time period from the data resource change record library, and output it as the first analysis cycle; The historical generation time point of the first analysis cycle is output as the first update time. If there is no first analysis period, the resource update demand coefficient of the current data resources will be combined with the predicted change data to calculate the first dynamic analysis period. The calculation formula for the first dynamic cycle is as follows: In the formula, This is represented as the first dynamic analysis period; This indicates the current preset dynamic update cycle; This is expressed as the resource update demand coefficient; This indicates setting an upper limit for the demand threshold; This represents the time interval between the predicted time point corresponding to the minimum change in the amount of metadata resources in the predicted change data and the current time. This represents the time interval between the predicted time point corresponding to the maximum change in metadata resource quantity in the predicted change data and the current time; e represents a constant with a value of 2.

7. This represents the weight of the impact of the current data resource update demand on the calculation of the first dynamic analysis cycle; This represents the weight of the impact of the current data resource change prediction on the calculation of the first dynamic analysis cycle; This represents the maximum change in the amount of metadata resources in the predicted change data; This is represented as the minimum change in the amount of metadata resources in the predicted change data; The first analysis dynamic cycle is output as the real-time dynamic cycle. If a single first analysis cycle exists, the second dynamic analysis cycle is calculated based on the historical time interval between the first update time of the current first analysis cycle and the current time, as well as the historical resource changes. The formula for calculating the second dynamic cycle is as follows: In the formula, This is represented as the second analysis dynamic cycle; This indicates the weight of the impact of resource demand and resource change forecasts on the calculation of the second analysis dynamic cycle; This represents the weight of the impact of historical updates of the current data resources on the calculation of the second analysis dynamic cycle. This represents the historical time interval between the first update moment of the current first analysis cycle and the current moment; This is represented as the predicted time period; It is expressed as the absolute difference between the amount of metadata resources predicted at the last moment and the amount of metadata resources at the current moment; This is expressed as the absolute difference between the amount of data resources at the first update time and the amount of data resources at the current time. The second analysis dynamic cycle is output as the real-time dynamic cycle. If there are multiple first analysis periods, the third dynamic analysis period is calculated based on the historical update interval between the first update times corresponding to all first analysis periods and the historical resource changes. The formula for calculating the third dynamic cycle is as follows: In the formula, This is represented as the third dynamic analysis cycle; This is represented as the c-th first analysis period, where c = 1, 2, , n; n represents the total number of the first analysis cycles; It is represented as the absolute difference between the amount of metadata resources at the first update time corresponding to the c-th first analysis period and the amount of metadata resources at the current time. The third analysis dynamic cycle is output as a real-time dynamic cycle.

Images