Metadata access method and device based on metadata model, equipment and medium
Through the automated access method based on the metadata model, the inefficiency and consistency of manual operation and file interaction methods in metadata access are solved, and the standardized processing of metadata is realized, and the accuracy and consistency of data are improved. Provides a reliable foundation for data analysis and application.
Patent Information
- Application Number
- CN202410928396.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
Smart Images

Figure CN118939608A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a metadata access method, device, computer equipment and storage medium based on a metadata model. Background Art
[0002] With the rapid development of information technology and the sharp increase in data volume, metadata management has become increasingly important. Metadata is data about data. It provides key information such as data description, source, quality, and authority, and is the basis for data management, search, integration, and sharing. General metadata access solutions include:
[0003] 1. Manually input metadata: In traditional metadata access solutions, a common way is to add metadata by manual input.
[0004] 2. Metadata access based on file interaction: Data providers export metadata in files of a specific format and provide them to data consumers. Data consumers then import these files into their own systems for parsing and processing.
[0005] In the above manual filling method, users or data administrators need to manually fill in various fields of metadata, such as title, description, keywords, etc., according to the actual situation of the data. Although this method is simple and direct, it is inefficient, prone to errors, and it is difficult to ensure the consistency and accuracy of the data.
[0006] The above file interaction method is more convenient than the manual method. Although this method can realize the sharing and exchange of metadata, it has the following problems: the file format is not unified and the parsing complexity is high; data loss or damage is prone to occur during file transmission; and it is difficult to ensure the real-time and consistency of data. Summary of the invention
[0007] Based on this, it is necessary to provide a metadata access method, device, computer equipment and storage medium based on a metadata model to address the above-mentioned technical problems. The method can automatically access metadata based on the metadata model without manual operation, and match the metadata dictionary and the metadata model, thereby solving the problems of inconsistent formats and high parsing complexity faced by file interaction methods.
[0008] A metadata access method based on a metadata model comprises: determining the category of metadata to be accessed in a database of a system to be accessed; constructing a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed; acquiring a metadata dictionary of the database of the system to be accessed; accessing the metadata to be accessed in the database of the system to be accessed; processing the accessed metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; and storing the metadata matching the metadata model.
[0009] In one of the embodiments, a metadata model is constructed based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, including: determining the data asset category, data asset source and data asset type based on the user usage demand information and the business usage demand information; based on the category of metadata to be accessed, instantiating according to the data asset category, data asset source and data asset type to obtain a metadata model.
[0010] In one of the embodiments, the category of metadata to be accessed is feature or data table; when the category of metadata to be accessed is feature, the metadata model obtained is a first metadata model, in which the data asset category is feature, the data asset source is a feature platform, and the data asset type is an offline feature or a real-time feature; when the category of metadata to be accessed is a data table, the metadata model obtained is a second metadata model, in which the data asset category is a data table, the data asset source is a data middle platform, and the data asset type is an original table of the operating data storage layer, an intermediate table of the general data model layer, or an application layer table.
[0011] In one embodiment, when the category of metadata to be accessed is a data table, the metadata matching the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables.
[0012] In one of the embodiments, when the category of metadata to be accessed is feature, the metadata dictionary of the database of the system to be accessed includes fields for feature processing logic and fields for feature output table, fields for feature coding name, and fields for technology or business; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the original table of the operation data storage layer, the metadata dictionary of the database of the system to be accessed includes fields for the data source system, fields for the source table name, fields for the person in charge of the source table, fields for the data synchronization method, and fields for frequency; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the intermediate table of the general data model layer, the metadata dictionary of the database of the system to be accessed includes fields for the logic of intermediate table processing, fields for table name, fields for conceptual model information, and fields for logical model information.
[0013] In one of the embodiments, storing metadata matching a metadata model includes: storing metadata matching the metadata model in a business database of a data directory; a metadata access method based on a metadata model also includes: constructing a mapping relationship between a metadata model and a metadata dictionary; synchronizing the mapping relationship between the metadata model and the metadata dictionary to the data directory; displaying the data directory, wherein a display interface of the data directory includes a data asset type management display, a metadata dictionary management display, and a metadata display, wherein the data asset type management display is used to display relevant information of the metadata model, the metadata dictionary management display is used to display relevant information of the metadata dictionary, and the metadata display is used to display relevant information of the metadata matching the metadata model.
[0014] In one of the embodiments, the metadata display includes a display module configuration and a display content configuration. The display module configuration includes basic information items, data overview items, data details items, and data lineage items. When any one or more of the basic information items, data overview items, data details items, and data lineage items are triggered and selected, the display content configuration displays relevant information of the metadata corresponding to the triggered selection item.
[0015] A metadata access device based on a metadata model comprises: a determination module for determining the category of metadata to be accessed in a database of a system to be accessed; a construction module for constructing a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed; an acquisition module for acquiring a metadata dictionary of the database of the system to be accessed; an access module for accessing the metadata to be accessed in the database of the system to be accessed; a processing module for processing the accessed metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; and a storage module for storing the metadata matching the metadata model.
[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above-mentioned embodiments are implemented.
[0017] A computer-readable storage medium stores a computer program, which implements the steps of any of the above-mentioned method embodiments when the computer program is executed by a processor.
[0018] The above-mentioned metadata access method, device, computer equipment and storage medium based on the metadata model determine the category of metadata to be accessed in the database of the system to be accessed, build a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, obtain the metadata dictionary of the database of the system to be accessed, access the metadata to be accessed in the database of the system to be accessed, process the accessed metadata according to the metadata model and the metadata dictionary, obtain metadata matching the metadata model, and store the metadata matching the metadata model.
[0019] Therefore, by constructing a metadata dictionary and a metadata model to standardize the incoming metadata, errors and inconsistencies that may occur during manual filling or file interaction can be effectively avoided, ensuring the accuracy and consistency of the data, and providing a reliable foundation for subsequent data analysis and application. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 is an application environment diagram of a metadata access method based on a metadata model in an embodiment;
[0021] Figure 2 A flowchart of a metadata access method based on a metadata model in an embodiment;
[0022] Figure 3 This is an interface diagram of a display interface of a data directory in a scenario in an embodiment;
[0023] Figure 4 This is an interface diagram of a display interface of a data directory in another scenario in an embodiment;
[0024] Figure 5 This is an interface diagram of a display interface in another scenario of a data directory in an embodiment;
[0025] Figure 6 A schematic diagram of a processing flow of a metadata access method based on a metadata model in an embodiment;
[0026] Figure 7 is a structural block diagram of a metadata access device based on a metadata model in an embodiment;
[0027] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0029] This application provides a metadata access method based on a metadata model, which is applied to Figure 1 In the application environment shown in FIG. 1 , the data platform 100 is used to execute a metadata access method based on a metadata model of the present application. The data platform 100 needs to access metadata in databases of multiple external systems. Figure 1 As shown, the databases of each external system include database 1, database 2, ... database n. The data platform 100 reads the corresponding metadata fields from each database and accesses the corresponding metadata. Specifically, when accessing the metadata of the database of the system to be accessed, the category of the metadata to be accessed is first determined. Then, the user usage demand information and / or business usage demand information is read from the internal database 200, and a metadata model is constructed based on the user usage demand information and / or business usage demand information and the category of the metadata to be accessed, and the metadata dictionary of the database of the system to be accessed is obtained. When the metadata to be accessed in the database of the system to be accessed is accessed, the accessed metadata is processed according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model, and the metadata matching the metadata model is stored in the internal database 200. Therefore, by constructing a metadata dictionary and a metadata model to standardize the accessed metadata, errors and inconsistencies that may occur during manual filling or file interaction are effectively avoided, the accuracy and consistency of the data are ensured, and a reliable foundation is provided for subsequent data analysis and application.
[0030] In one embodiment, Figure 2 As shown, a metadata access method based on a metadata model is provided, and the method is applied to Figure 1 The data platform 100 in FIG. 1 is used as an example to illustrate the method, which includes the following steps:
[0031] S202: Determine the category of metadata to be accessed in the database of the system to be accessed.
[0032] In this embodiment, the data platform performs classified access management on the accessed data assets. That is, metadata is accessed to the data platform according to categories. For example, the categories of metadata include features, data tables, and indicators.
[0033] S204: construct a metadata model based on the user usage requirement information and / or the business usage requirement information and the category of the metadata to be accessed.
[0034] The accessed metadata has a certain data structure, but the metadata structures stored in the databases of various systems are different. That is, the metadata accessed to the data platform is non-standard and non-uniform. When storing such metadata, a model structure is abstracted, and the metadata of each accessed system is processed through the model structure to obtain standard and unified metadata. In this embodiment, when constructing a metadata model, it is necessary to consider user usage requirements and business usage requirements. User usage requirements are obtained through user demand surveys, and business usage requirements are obtained through business analysis. Specifically, based on the actual business and user demands, user usage demand information and / or business usage demand information is abstracted, the required metadata structure is refined, and then combined with the specific categories of metadata, a corresponding metadata model is constructed.
[0035] In one embodiment, the above-mentioned metadata model is constructed based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, including: determining the data asset category, data asset source and data asset type based on the user usage demand information and / or business usage demand information; based on the category of metadata to be accessed, instantiating according to the data asset category, data asset source and data asset type to obtain the metadata model.
[0036] In this embodiment, user usage demand information is determined by investigating user needs. After the user usage demand information is converted into business, the business is analyzed to obtain business usage demand information. The data asset category, data asset source and data asset type are determined based on the user usage demand information and / or the business usage demand information. It can be: when the business usage demand information and the user usage demand information exist at the same time, the data asset category, data asset source and data asset type are determined only based on the business usage demand information. Therefore, there is no need to analyze the user usage demand again, and the data asset category, data asset source and data asset type can be determined through the converted business usage demand information. Or, when the business usage demand information and the user usage demand information exist at the same time, the data asset category, data asset source and data asset type are determined based on the business usage demand information and the user usage demand information. Therefore, when only one or two of the data asset category, data asset source and data asset type can be determined based on the business usage demand information, the user usage demand information is introduced to determine the remaining one or two. That is, if there are omissions in the information conversion when converting the user usage demand information into the business usage demand information, the user usage demand information is used to assist in correcting the final result and provide the accuracy of the final data asset category, data asset source and data asset type. Or, when there is only user usage demand information, the data asset category, data asset source and data asset type are determined based on the user usage demand information. Therefore, in the case of no business usage demand information conversion, the user usage demand information can be directly used to determine the final result.
[0037] Specifically, the user usage demand information and / or business usage demand information is abstracted, the required metadata structure is extracted, and then the data asset category, data asset source, and data asset type are determined based on the extracted metadata structure. For example, the extracted metadata structure can be divided into content structure, semantic structure, and syntactic structure, and the data asset category, data asset source, and data asset type are determined from the metadata structure in the actual business scenario. Then, based on the specific category of metadata to be accessed, instantiation is performed according to the data asset category, data asset source, and data asset type to obtain a metadata model.
[0038] In one example, the category of metadata to be accessed is feature or data table; when the category of metadata to be accessed is feature, the metadata model obtained is a first metadata model, in which the data asset category is feature, the data asset source is a feature platform, and the data asset type is an offline feature or a real-time feature; when the category of metadata to be accessed is a data table, the metadata model obtained is a second metadata model, in which the data asset category is a data table, the data asset source is a data middle platform, and the data asset type is an original table of the operating data storage layer, an intermediate table of the general data model layer, or an application layer table.
[0039] Specifically, the data platform provides data asset category management. Data asset categories include features and data tables. Data asset sources include feature platforms and data middle platforms. In data asset types: If the data asset category is a feature, the data asset type can be an offline feature or a real-time feature, etc. If the data asset category is a data table, the data asset type can be an ODS (Operational Data Store) original table, a CDM (Common Data Model) intermediate table, or an application layer table, etc.
[0040] When accessing metadata in the database of other systems, first determine the category of the metadata, and then determine the metadata model based on the category. For example, if the category of the accessed metadata is feature, then determine the first metadata model. The first metadata model is represented by the data asset category as feature, the data asset source as feature platform, and the data asset type as offline feature.
[0041] In one example, when the category of metadata to be accessed is data table, the metadata matching the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables.
[0042] Specifically, data asset types, data asset numbers, data asset attributes, operations, relationships, etc. are abstracted based on the survey of user usage requirements and business usage requirements. Then, specific data assets are instantiated. For example, data assets are metadata, and the category of metadata is data tables. The metadata that is finally determined to match the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables. Among them, the attributes of the data table may include the English name of the data table, the Chinese name of the data table, and the creation time of the data table. The operation class for creating a data table includes the user who creates the data table or the user who modifies the data table. The relationship between the data tables includes the association keys of the data tables.
[0043] In one example, when the category of metadata to be accessed is feature, the metadata dictionary of the database of the system to be accessed includes fields for feature processing logic and fields for feature output table, fields for feature coding name, and fields for technology or business; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the original table of the operation data storage layer, the metadata dictionary of the database of the system to be accessed includes fields for the data source system, fields for the source table name, fields for the person in charge of the source table, fields for the method of data synchronization, and fields for frequency; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the intermediate table of the general data model layer, the metadata dictionary of the database of the system to be accessed includes fields for the logic of intermediate table processing, fields for table name, fields for conceptual model information, and fields for logical model information.
[0044] Specifically, when the data platform accesses metadata, it first maintains the management information of the metadata category on the system, selects the specific data asset category, data asset source and data asset type, and records the metadata dictionary of this category. For example, for the feature category, the data fields in the corresponding metadata dictionary include feature processing logic, feature output table, feature coding name and person in charge and other technical or business metadata information. The data fields in the metadata dictionary corresponding to the ODS original table include information such as the data source system, source table name, source table person in charge, data synchronization method and frequency. The data fields in the metadata dictionary corresponding to the CDM intermediate table include the logic of intermediate table processing, table name, conceptual model information and logical model information.
[0045] S206, obtaining a metadata dictionary of the database to be connected to the system.
[0046] In this embodiment, when all metadata of the database of the system to be accessed are accessed, a corresponding metadata dictionary is constructed. The metadata dictionary integrates the metadata of the system to be accessed and also provides a basis for its subsequent standardized processing. Specifically, when storing metadata, the database of the system to be accessed uses a metadata dictionary to manage the stored metadata. For example, multiple metadata of various tables are stored in the database of the system to be accessed. Among them, the multiple metadata may include table name, table description information, user who created the table and table creation time, table modification operation and modification time and user of modification operation, etc. These metadata are abstracted to obtain a metadata dictionary. The metadata dictionary includes fields such as table name, table description, creation time, creator, creation time, updater, update time, and update operation.
[0047] S208, accessing the metadata to be accessed in the database of the system to be accessed.
[0048] In this embodiment, all metadata in the database of each system to be connected are connected to the internal database that carries the metadata. The database can be selected according to actual needs, such as MySQL database.
[0049] S210: Process the received metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model.
[0050] In this embodiment, the metadata model is an abstraction of the form of metadata storage. The metadata dictionary contains all metadata attributes and is used to assist users in finding and understanding the data structure in the database. According to the metadata model, the metadata can be actually stored according to a certain model structure, and the corresponding metadata can be identified and found according to the metadata dictionary.
[0051] For example, if the metadata to be accessed is metadata of a feature type, a metadata model is constructed for the feature, and then the corresponding metadata is searched according to the metadata dictionary, and stored according to the metadata model of the feature, and then the metadata of the feature is stored in the internal database.
[0052] S212: storing metadata matching the metadata model.
[0053] In this embodiment, metadata is stored in a manner that matches the metadata model, so that the stored metadata meets the unified standard requirements of the system. That is, for different systems with different metadata storage structures, this embodiment stores the metadata of each system in a manner that matches the metadata model, so that when metadata of multiple systems are accessed, the standardization and uniformity of metadata storage can be ensured.
[0054] The above-mentioned metadata access method based on the metadata model determines the category of metadata to be accessed in the database of the system to be accessed, builds a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, obtains the metadata dictionary of the database of the system to be accessed, accesses the metadata to be accessed in the database of the system to be accessed, processes the accessed metadata according to the metadata model and the metadata dictionary, obtains metadata matching the metadata model, and stores the metadata matching the metadata model. Therefore, by constructing a metadata dictionary and a metadata model to standardize the accessed metadata, errors and inconsistencies that may occur during manual filling or file interaction are effectively avoided, the accuracy and consistency of the data are ensured, and a reliable foundation is provided for subsequent data analysis and application.
[0055] In one embodiment, the above-mentioned storing metadata matching the metadata model includes: storing the metadata matching the metadata model in the business database of the data directory; after the above-mentioned step of instantiating the metadata model based on the category of the metadata to be accessed, according to the data asset category, the data asset source and the data asset type, the metadata model is obtained, it also includes: constructing a mapping relationship between the metadata model and the metadata dictionary; synchronizing the mapping relationship between the metadata model and the metadata dictionary to the data directory; displaying the data directory, the display interface of the data directory includes a data asset type management display, a metadata dictionary management display and a metadata display, the data asset type management display is used to display relevant information of the metadata model, the metadata dictionary management display is used to display relevant information of the metadata dictionary, and the metadata display is used to display relevant information of the metadata matching the metadata model.
[0056] Among them, the metadata display includes display module configuration and display content configuration. The display module configuration includes basic information items, data overview items, data details items and data lineage items. When any one or more of the basic information items, data overview items, data details items and data lineage items are triggered and selected, the display content configuration displays relevant information of the metadata corresponding to the triggered selection item.
[0057] In this embodiment, after the mapping relationship between the metadata model and the metadata dictionary is constructed, the mapping relationship between the metadata model and the metadata dictionary is synchronized to the data directory. Figure 3 As shown, after constructing the metadata model according to the asset category, asset source and asset type of the metadata, the corresponding metadata dictionary can be retrieved based on the mapping relationship between the metadata model and the metadata dictionary. For example, in the constructed metadata model, the asset category is feature, the asset source is the feature platform, and the asset type is offline feature. Based on the mapping relationship, its metadata dictionary can be retrieved and then added to the display interface of the data directory. Specifically, after the system maintains the metadata dictionary and metadata model, the person in charge of each type of data asset processes the metadata according to the metadata dictionary and synchronizes it to the business database of the data directory. In addition, set the fields of metadata that need to be displayed for each type of data asset. For example: feature-type data assets choose which metadata fields to display and sort, etc.
[0058] like Figure 3 As shown in the figure, the display interface of the data directory includes directory management, classification management, tag management, asset type management, metadata dictionary management, and metadata display configuration. Figure 4 As shown in the figure, the metadata dictionary management displays the field name, field Chinese name, field description, creator, and creation time corresponding to the metadata model. Figure 5As shown in , metadata display configuration includes display module configuration and display content configuration. Display module configuration includes basic information items, data overview items, data details items, and data lineage items. The data details items can be divided into table structure items and data sample items. Each item in the display module configuration can be triggered. Figure 5 As shown in the figure, the basic information item and the data overview item are selected. At this time, the display content configuration displays the detailed information of the basic item and the detailed information of the data overview item respectively. Figure 5 As shown, the basic item is triggered in the display content configuration, and the detailed information of the basic item is displayed. The detailed information of the basic item includes the specific field name, the Chinese name of the field, the display name, and the prompt copy. Therefore, after the access to metadata in the data asset is standardized through the above embodiment, the metadata field can be displayed on demand in the data directory according to the maintained metadata dictionary.
[0059] In summary, the metadata access method based on the metadata model proposed in this application proposes a two-layer design based on the form of a database. The bottom layer directly accesses all metadata in the databases of external systems and forms a metadata dictionary for each system. Then, the metadata model abstracted according to user usage requirements and actual business usage requirements is automatically matched to obtain a standardized metadata storage structure, and then the metadata storage structure is used to display data in various forms. For the specific processing flow, see Figure 6 As shown, the following steps are included:
[0060] Step 1: Abstract based on actual business and user demands, refine the structure of required metadata, and build a metadata data model.
[0061] Step 2: Connect all metadata of each system to the database that carries metadata information. You can choose a database such as MySQL according to actual needs, and form a metadata dictionary for each system at the access layer. At this point, the metadata already has a certain structure, but it is not standardized and unified.
[0062] Step 3: Based on the metadata model of the metadata and the metadata dictionary of the access layer, use the big model tool to automatically match, and then perform simple proofreading and adjustment manually to form a standard mapping relationship.
[0063] Step 4: After the above steps, the relevant mapping relationship has been clarified. The metadata can be stored in the standard layer according to the standardized metadata storage structure, and the data can be displayed as needed.
[0064] In summary, compared with the prior art, the metadata access method based on the metadata model proposed in this application has the following advantages:
[0065] Advantage 1: Improve metadata processing efficiency
[0066] This application uses an automated metadata access and management method, especially using metadata models for automatic matching, which greatly reduces the need for manual intervention. Compared with manual filling or file interaction, this method significantly improves the efficiency of metadata processing and reduces labor and time costs.
[0067] Advantage 2: Enhanced data accuracy and consistency
[0068] This application constructs a metadata dictionary and uses a unified metadata model to standardize the metadata that is accessed, effectively avoiding errors and inconsistencies that may occur during manual filling or file interaction. The standardized storage structure ensures the accuracy and consistency of the data, providing a reliable foundation for subsequent data analysis and application.
[0069] Advantage 3: Improving the flexibility and scalability of data management
[0070] The two-layer technical design of this application: the underlying metadata dictionary and the upper standardized metadata storage structure make data management more flexible. At the same time, this design also facilitates the expansion and management of metadata in the future, meeting the needs of business development and data growth.
[0071] Advantage 4: Ensuring data integrity and security
[0072] Compared with the file interaction method, this application manages metadata based on the database, effectively avoiding the data loss or damage that may occur during file transmission. The data integrity and security mechanisms of the database itself, such as backup, recovery, and access control, further ensure the integrity and security of metadata.
[0073] Advantage 5: Optimize data display and user experience
[0074] Through the standardized metadata storage structure, this application makes data display more flexible and diverse. Users can query, analyze and visualize data on demand, thereby optimizing data display effects and user experience. This customized data display method meets the needs of different users and businesses and enhances the actual application value of data.
[0075] It should be understood that, although the steps in the flowchart are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0076] The present application also provides a metadata access device based on a metadata model. Figure 7 As shown, a metadata access device based on a metadata model includes a determination module 702, a construction module 704, an acquisition module 706, an access module 708, a processing module 710 and a storage module 712. The determination module 702 is used to determine the category of metadata to be accessed in the database of the system to be accessed; the construction module 704 is used to construct a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed; the acquisition module 706 is used to obtain the metadata dictionary of the database of the system to be accessed; the access module 708 is used to access the metadata to be accessed in the database of the system to be accessed; the processing module 710 is used to process the accessed metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; and the storage module 712 is used to store the metadata matching the metadata model.
[0077] In one of the embodiments, a metadata model is constructed based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, including: determining the data asset category, data asset source and data asset type based on the user usage demand information and the business usage demand information; based on the category of metadata to be accessed, instantiating according to the data asset category, data asset source and data asset type to obtain a metadata model.
[0078] In one of the embodiments, the category of metadata to be accessed is feature or data table; when the category of metadata to be accessed is feature, the metadata model obtained is a first metadata model, in which the data asset category is feature, the data asset source is a feature platform, and the data asset type is an offline feature or a real-time feature; when the category of metadata to be accessed is a data table, the metadata model obtained is a second metadata model, in which the data asset category is a data table, the data asset source is a data middle platform, and the data asset type is an original table of the operating data storage layer, an intermediate table of the general data model layer, or an application layer table.
[0079] In one embodiment, when the category of metadata to be accessed is a data table, the metadata matching the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables.
[0080] In one of the embodiments, when the category of metadata to be accessed is feature, the metadata dictionary of the database of the system to be accessed includes fields for feature processing logic and fields for feature output table, fields for feature coding name, and fields for technology or business; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the original table of the operation data storage layer, the metadata dictionary of the database of the system to be accessed includes fields for the data source system, fields for the source table name, fields for the person in charge of the source table, fields for the data synchronization method, and fields for frequency; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the intermediate table of the general data model layer, the metadata dictionary of the database of the system to be accessed includes fields for the logic of intermediate table processing, fields for table name, fields for conceptual model information, and fields for logical model information.
[0081] In one of the embodiments, storing metadata matching a metadata model includes: storing metadata matching the metadata model in a business database of a data directory; a metadata access device based on a metadata model also includes a display module for building a mapping relationship between the metadata model and the metadata dictionary; synchronizing the mapping relationship between the metadata model and the metadata dictionary to the data directory; displaying the data directory, wherein the display interface of the data directory includes a data asset type management display, a metadata dictionary management display, and a metadata display, wherein the data asset type management display is used to display relevant information of the metadata model, the metadata dictionary management display is used to display relevant information of the metadata dictionary, and the metadata display is used to display relevant information of the metadata matching the metadata model.
[0082] In one of the embodiments, the metadata display includes a display module configuration and a display content configuration. The display module configuration includes basic information items, data overview items, data details items, and data lineage items. When any one or more of the basic information items, data overview items, data details items, and data lineage items are triggered and selected, the display content configuration displays relevant information of the metadata corresponding to the triggered selection item.
[0083] For the specific definition of a metadata access device based on a metadata model, please refer to the definition of a metadata access method based on a metadata model in the above text, which will not be repeated here. Each module in the above-mentioned metadata access device based on a metadata model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0084] In one embodiment, a computer device is provided. The computer device may be a server supporting the operation of the data platform 100. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to interact with an external system for data, read metadata in a database of an external system, etc. When the computer program is executed by the processor, the above-mentioned metadata access method based on the metadata model is implemented.
[0085] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: determining the category of metadata to be accessed in a database of a system to be accessed; constructing a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed; acquiring a metadata dictionary of the database of the system to be accessed; accessing the metadata to be accessed in the database of the system to be accessed; processing the accessed metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; and storing the metadata matching the metadata model.
[0087] In one of the embodiments, when the processor executes a computer program to implement the above-mentioned step of constructing a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, the following steps are specifically implemented: based on the user usage demand information and the business usage demand information, the data asset category, data asset source and data asset type are determined; based on the category of metadata to be accessed, the data asset category, data asset source and data asset type are instantiated to obtain a metadata model.
[0088] In one of the embodiments, the category of metadata to be accessed is feature or data table; when the category of metadata to be accessed is feature, the metadata model obtained is a first metadata model, in which the data asset category is feature, the data asset source is a feature platform, and the data asset type is an offline feature or a real-time feature; when the category of metadata to be accessed is a data table, the metadata model obtained is a second metadata model, in which the data asset category is a data table, the data asset source is a data middle platform, and the data asset type is an original table of the operating data storage layer, an intermediate table of the general data model layer, or an application layer table.
[0089] In one embodiment, when the category of metadata to be accessed is a data table, the metadata matching the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables.
[0090] In one of the embodiments, when the category of metadata to be accessed is feature, the metadata dictionary of the database of the system to be accessed includes fields for feature processing logic and fields for feature output table, fields for feature coding name, and fields for technology or business; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the original table of the operation data storage layer, the metadata dictionary of the database of the system to be accessed includes fields for the data source system, fields for the source table name, fields for the person in charge of the source table, fields for the data synchronization method, and fields for frequency; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the intermediate table of the general data model layer, the metadata dictionary of the database of the system to be accessed includes fields for the logic of intermediate table processing, fields for table name, fields for conceptual model information, and fields for logical model information.
[0091] In one of the embodiments, when the processor executes a computer program to implement the above-mentioned step of storing metadata matching the metadata model, the following steps are specifically implemented: storing the metadata matching the metadata model in a business database of the data directory; when the processor executes the computer program, the following steps are implemented: building a mapping relationship between the metadata model and the metadata dictionary; synchronizing the mapping relationship between the metadata model and the metadata dictionary to the data directory; displaying the data directory, the display interface of the data directory includes a data asset type management display, a metadata dictionary management display and a metadata display, the data asset type management display is used to display relevant information of the metadata model, the metadata dictionary management display is used to display relevant information of the metadata dictionary, and the metadata display is used to display relevant information of the metadata matching the metadata model.
[0092] In one of the embodiments, the metadata display includes a display module configuration and a display content configuration. The display module configuration includes basic information items, data overview items, data details items, and data lineage items. When any one or more of the basic information items, data overview items, data details items, and data lineage items are triggered and selected, the display content configuration displays relevant information of the metadata corresponding to the triggered selection item.
[0093] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: determining the category of metadata to be accessed in a database of a system to be accessed; building a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed; acquiring a metadata dictionary of the database of the system to be accessed; accessing the metadata to be accessed in the database of the system to be accessed; processing the accessed metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; and storing the metadata matching the metadata model.
[0094] In one of the embodiments, when the computer program is executed by the processor to implement the above-mentioned step of constructing a metadata model based on user usage demand information and / or business usage demand information and the category of metadata to be accessed, the following steps are specifically implemented: based on the user usage demand information and the business usage demand information, the data asset category, data asset source and data asset type are determined; based on the category of the metadata to be accessed, the data asset category, data asset source and data asset type are instantiated to obtain the metadata model.
[0095] In one of the embodiments, the category of metadata to be accessed is feature or data table; when the category of metadata to be accessed is feature, the metadata model obtained is a first metadata model, in which the data asset category is feature, the data asset source is a feature platform, and the data asset type is an offline feature or a real-time feature; when the category of metadata to be accessed is a data table, the metadata model obtained is a second metadata model, in which the data asset category is a data table, the data asset source is a data middle platform, and the data asset type is an original table of the operating data storage layer, an intermediate table of the general data model layer, or an application layer table.
[0096] In one embodiment, when the category of metadata to be accessed is a data table, the metadata matching the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables.
[0097] In one of the embodiments, when the category of metadata to be accessed is feature, the metadata dictionary of the database of the system to be accessed includes fields for feature processing logic and fields for feature output table, fields for feature coding name, and fields for technology or business; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the original table of the operation data storage layer, the metadata dictionary of the database of the system to be accessed includes fields for the data source system, fields for the source table name, fields for the person in charge of the source table, fields for the data synchronization method, and fields for frequency; when the category of metadata to be accessed is data table and the data asset type in the second metadata model is the intermediate table of the general data model layer, the metadata dictionary of the database of the system to be accessed includes fields for the logic of intermediate table processing, fields for table name, fields for conceptual model information, and fields for logical model information.
[0098] In one of the embodiments, when the computer program is executed by the processor to implement the above-mentioned step of storing the metadata matching the metadata model, the following steps are specifically implemented: storing the metadata matching the metadata model in the business database of the data directory; when the computer program is executed by the processor, the following steps are implemented: building a mapping relationship between the metadata model and the metadata dictionary; synchronizing the mapping relationship between the metadata model and the metadata dictionary to the data directory; displaying the data directory, the display interface of the data directory includes a data asset type management display, a metadata dictionary management display and a metadata display, the data asset type management display is used to display relevant information of the metadata model, the metadata dictionary management display is used to display relevant information of the metadata dictionary, and the metadata display is used to display relevant information of the metadata matching the metadata model.
[0099] In one of the embodiments, the metadata display includes a display module configuration and a display content configuration. The display module configuration includes basic information items, data overview items, data details items, and data lineage items. When any one or more of the basic information items, data overview items, data details items, and data lineage items are triggered and selected, the display content configuration displays relevant information of the metadata corresponding to the triggered selection item.
[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0101] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0102] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A metadata access method based on a metadata model, It is characterized in that The method comprises: Determine the category of metadata to be accessed in the database of the system to be accessed; Building a metadata model based on the user usage demand information and / or the business usage demand information and the category of the metadata to be accessed; Obtaining a metadata dictionary of the database of the system to be accessed; Accessing the metadata to be accessed in the database of the system to be accessed; Processing the accessed metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; The metadata matching the metadata model is stored.
2. The method according to claim 1, It is characterized in that The step of constructing a metadata model based on the user usage requirement information and / or the business usage requirement information and the category of the metadata to be accessed includes: Determine the data asset category, data asset source and data asset type based on the user usage demand information and the business usage demand information; Based on the category of the metadata to be accessed, instantiation is performed according to the data asset category, the data asset source, and the data asset type to obtain the metadata model.
3. The method according to claim 2, It is characterized in that The category of the metadata to be accessed is a feature or a data table; When the category of the metadata to be accessed is a feature, the obtained metadata model is a first metadata model, in which the data asset category is a feature, the data asset source is a feature platform, and the data asset type is an offline feature or a real-time feature; When the category of the metadata to be accessed is a data table, the metadata model obtained is a second metadata model, in which the data asset category is a data table, the data asset source is a data middle platform, and the data asset type is an original table of the operating data storage layer, an intermediate table of the general data model layer, or an application layer table.
4. The method according to claim 3, It is characterized in that When the category of the metadata to be accessed is a data table, the metadata matching the metadata model includes the number of the data table, the type of the data table, the attributes of the data table, the operation class for creating the data table, and the relationship between the data tables.
5. The method according to claim 3, It is characterized in that When the category of the metadata to be accessed is feature, the metadata dictionary of the database of the system to be accessed includes a field of feature processing logic and a field of feature output table, a field of feature code name, and a field of technology or business; When the category of the metadata to be accessed is a data table and the data asset type in the second metadata model is an original table of the operation data storage layer, the metadata dictionary of the database of the system to be accessed includes a field of the data source system, a field of the source table name, a field of the source table owner, a field of the data synchronization method, and a field of the frequency; When the category of the metadata to be accessed is a data table and the data asset type in the second metadata model is an intermediate table of the general data model layer, the metadata dictionary of the database of the system to be accessed includes logical fields processed by the intermediate table, table name fields, conceptual model information fields, and logical model information fields.
6. The method according to claim 2, It is characterized in that The storing the metadata matching the metadata model includes: storing the metadata matching the metadata model in a business database of a data directory; The method further comprises: Constructing a mapping relationship between the metadata model and the metadata dictionary; Synchronizing the mapping relationship between the metadata model and the metadata dictionary to the data directory; The data directory is displayed. The display interface of the data directory includes a data asset type management display, a metadata dictionary management display and a metadata display. The data asset type management display is used to display relevant information of the metadata model, the metadata dictionary management display is used to display relevant information of the metadata dictionary, and the metadata display is used to display relevant information of the metadata matching the metadata model.
7. The method according to claim 6, It is characterized in that The metadata display includes display module configuration and display content configuration. The display module configuration includes basic information items, data overview items, data details items and data lineage items. When any one or more of the basic information items, the data overview items, the data details items and the data lineage items are triggered and selected, the display content configuration displays relevant information of the metadata corresponding to the triggered selection item.
8. A metadata access device based on a metadata model, It is characterized in that The device comprises: A determination module, used to determine the category of metadata to be accessed in the database of the system to be accessed; A construction module, used to construct a metadata model based on user usage demand information and / or business usage demand information and the category of the metadata to be accessed; An acquisition module, used for acquiring a metadata dictionary of the database of the system to be connected; An access module, used for accessing the metadata to be accessed in the database of the system to be accessed; A processing module, used to process the received metadata according to the metadata model and the metadata dictionary to obtain metadata matching the metadata model; A storage module is used to store the metadata matching the metadata model.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Metadata-based big data platform construction method, system, equipment and medium
CN113051263A
Method for importing Excel table data into database based on metadata definition and implementation
CN115934716A
College relational data processing method and system based on lake and warehouse integration
CN117112667A
Importing data using metadata
US6993529B1
