Low-delay consistency metadata retrieval method

Abstract

The invention discloses a low-delay consistency metadata retrieval method, and relates to the technical field of metadata management. The low-delay consistency metadata retrieval method comprises the following steps that a front end retrieves metadata through a central metadata retrieval interface, and the central metadata retrieval interface finds an adaptive metadata plug-in through metadata plug-in management to obtain corresponding technical metadata; the central metadata retrieval interface obtains management metadata and business metadata by querying the metadata management system DB, and the central metadata retrieval interface summarizes the metadata and returns the summarized metadata to the front end. According to the scheme, metadata retrieval is carried out based on the central metadata retrieval interface, data assembly is carried out in the retrieval interface, the assembled metadata is returned to the front end for rendering, the consistency of the metadata and the timeliness of metadata change acquisition are guaranteed, information errors caused by metadata inconsistency are reduced through the scheme, and the efficiency is improved. And information lag caused by untimely change and incapability of obtaining information is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of metadata management, and in particular to a low-latency consistency metadata retrieval method. Background Art

[0002] At present, the function of search is to search and give the content that users want most under the premise of a given keyword. However, with the development of cloud application technology, more and more applications are deployed in the cloud. At the same time, with the increase in the number of users, the data of applications is also increasing. Simply using the search function in the database not only has poor results, but also affects the use of other functions of the database. For this reason, developers generally use independent servers to build search services separately.

[0003] Metadata, also known as intermediate data or relay data, is data that describes data. It mainly contains attribute information about the data and is used to support a variety of functions, such as indicating storage locations, historical data, resource searches, and file records. Metadata can be considered an electronic catalog, the purpose of which is to assist data retrieval by describing and collecting the content or features of the data.

[0004] However, currently available similar technologies generally synchronize the source of factual metadata to the metadata system for storage of technical metadata through metadata synchronization. It is difficult to ensure the consistency between the source of factual metadata and the retrieved technical metadata. Changes to the source of factual metadata need to be updated to the metadata management system through the next scheduled synchronization before they can be seen in the search. The consistency is poor, and it takes a long time for the change in the source of factual metadata to be seen in the search. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a low-latency consistency metadata retrieval method, which solves the problems raised in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a low-latency consistency metadata retrieval method, comprising the following steps:

[0007] S1: The metadata is divided into technical metadata, business metadata, and management metadata. The business metadata is stored in the metadata management system DB by entering the metadata through a manual interface, and the management metadata is stored in the metadata management system DB by entering the metadata through a manual interface;

[0008] S2: the front end retrieves metadata through a central metadata retrieval interface;

[0009] S3: The central metadata retrieval interface finds the adapted metadata plug-in through the metadata plug-in management to obtain the corresponding technical metadata;

[0010] S4: The central metadata retrieval interface obtains management metadata and business metadata by querying the metadata management system DB;

[0011] S5: The central metadata retrieval interface summarizes the technical metadata, business metadata, and management metadata and returns them to the front end.

[0012] Business metadata refers to the business attributes of data, such as business definitions and business rules.

[0013] Technical metadata describes the structured data and the technical attributes of the data, such as the location where the data is stored, the type of field, the length of the field, etc.

[0014] Operational metadata refers to the description of the operational attributes of data, such as the data management department, the person responsible for data management, the data update frequency, etc.

[0015] A further improvement of the technical solution of the present invention is that the central metadata retrieval interface dynamically assembles various metadata, and the assembled metadata is returned to the front end for rendering in a standard return format. Various metadata are organized in parallel and enter the front end through a preset channel.

[0016] A further improvement of the technical solution of the present invention lies in: the metadata management architecture of the metadata involved in S1-S6 and the source of the factual metadata, the metadata management architecture consists of a front end connected to the central metadata retrieval interface, an APIController connected to the front end and serializing the metadata into the client request format, a Service Layer connected to the APIController, and a metadata plug-in Manager connected to the Service Layer and used to obtain metadata.

[0017] The corresponding interface is used to pass the retrieval metadata. In some scenarios, too many fields will affect performance. The metadata plug-in Manage can load some fields of the required metadata. The specific code is as follows:

[0018]

[0019] The corresponding interface is used to transfer the retrieval metadata. In some scenarios, too many fields will affect the performance. The metadata plug-in Manage can load some fields of the required metadata.

[0020] A further improvement of the technical solution of the present invention is that the metadata plug-in Manager is connected to MySQL, Dor is and Hi ve which are the sources of factual metadata, and the dynamic assembly of various types of metadata in the central metadata retrieval interface also includes the sources of factual metadata belonging to MySQL, Dor is and Hi ve.

[0021] A further improvement of the technical solution of the present invention is that a CDC module is set between the fact metadata source and the central metadata retrieval interface, and the CDC module realizes the configuration change tracking of the databases of MySQL, Dor is and Hi ve fact metadata sources. In order to avoid data redundancy, automatic cleanup of data with a retention period of two days is started, and a change tracking function is provided for recording the insertion, update and deletion operations of the tracked table.

[0022] A further improvement of the technical solution of the present invention is that the CDC module synchronizes the fact metadata source to the message middleware in real time, obtains metadata changes in a timely manner by subscribing to messages, and maintains data consistency through exactly-once semantics.

[0023] A further improvement of the technical solution of the present invention is that: the CDC module includes but is not limited to INSERT, UPDATE, and DELETE of fact metadata.

[0024] Among them, INSERT, UPDATE, and DELETE represent insert, update, and delete. Most of the modifications in fact metadata changes are a combination of insert, update, and delete.

[0025] A further improvement of the technical solution of the present invention is that the message middleware searches for the channel corresponding to the matching middleware according to the identifier of the source of the factual metadata, stores it, determines the path corresponding to the channel according to the channel, transmits the message to the queue corresponding to the channel, and performs retrieval after accepting the query request of the central metadata retrieval interface.

[0026] The execution script to start change tracking for the database is as follows:

[0027] ALTER DATABASE AdventureWorks

[0028] SET CHANGE TRACK IN = ON;

[0029] To avoid data redundancy, the execution script for starting automatic cleanup of data with a two-day data retention period is as follows:

[0030] {AUTO_CLEANUP=ON,CHANGE_RETENT ION=2DAYS}

[0031] ALTER TABLE dbo.Product

[0032] ENABLE CHANGE TRACKI NG;

[0033] The change tracking function is used to record the insert, update, and delete operations of the tracked table. The change summary functions used are as follows:

[0034] CHANGETABLE(CHANGES) – returns tracking information for all changes made to a table since the specified version;

[0035] CHANGETABLE(VERSION) – returns the latest change tracking information for the specified row;

[0036] CHANGE_TRACKI NG_M I N_VAL ID_VERS ION() – Returns the lowest valid version used to obtain change tracking information from the specified table when using the CHANGETABLE function;

[0037] CHANGE_TRACKI NG_CURRENT_VERS ION - Gets the version associated with the last committed transaction, which can be used the next time you enumerate changes using CHANGETABLE;

[0038] CHANGE_TRACKI NG_IS_COLUMN_I N_MASK—Used to interpret the SYS_CHANGE_COLUMNS values ​​returned by the CHANGETABLE{CHANGES...} function.

[0039] WITH CHANGE_TRACKING_CONTEXT—Used to specify the change context, such as the originator ID, when an application changes data.

[0040] Compared with the prior art, the beneficial effects of the present invention are: metadata retrieval is performed based on a central metadata retrieval interface, various types of metadata are dynamically assembled in the retrieval interface, and then the assembled metadata is returned to the front end for rendering in a standard return format to ensure metadata consistency and metadata change acquisition timeliness. This solution can reduce information errors caused by inconsistent metadata and information lags caused by untimely changes and inability to acquire information;

[0041] By synchronizing fact metadata sources to the message middleware in real time through CDC, obtaining metadata changes in a timely manner through subscription messages, and ensuring data consistency through exactly-once semantics, the data interface retrieval performance will be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a retrieval method in a low-latency consistency metadata retrieval method;

[0043] Figure 2 It is a retrieval schematic diagram of a central metadata retrieval interface in a low-latency consistency metadata retrieval method;

[0044] Figure 3 A schematic diagram of metadata acquisition in a low-latency consistency metadata retrieval method;

[0045] Figure 4 A flowchart of a metadata management architecture in a low-latency consistent metadata retrieval method;

[0046] Figure 5 The figure is a flowchart of the CDC module in the low-latency consistency metadata retrieval method. DETAILED DESCRIPTION

[0047] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0048] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0049] In addition, in order to better illustrate the present application, numerous specific details are provided in the specific embodiments below. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, and elements well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0050] Reference Figure 1-Figure 5 The present invention provides a low-latency consistency metadata retrieval method, comprising the following steps:

[0051] S1: The metadata is divided into technical metadata, business metadata, and management metadata. The business metadata is stored in the metadata management system DB by entering the metadata through a manual interface, and the management metadata is stored in the metadata management system DB by entering the metadata through a manual interface;

[0052] S2: the front end retrieves metadata through a central metadata retrieval interface;

[0053] S3: The central metadata retrieval interface finds the adapted metadata plug-in through the metadata plug-in management to obtain the corresponding technical metadata;

[0054] S4: The central metadata retrieval interface obtains management metadata and business metadata by querying the metadata management system DB;

[0055] S5: The central metadata retrieval interface summarizes the technical metadata, business metadata, and management metadata and returns them to the front end.

[0056] The fact metadata source in this technical solution refers to the metadata source of the external system data obtained, such as the metadata of the data stored in the MySQL library of the external system;

[0057] Business metadata refers to the business attributes of data, such as business definitions and business rules.

[0058] Technical metadata describes the structured data and the technical attributes of the data, such as the location where the data is stored, the type of the field, the length of the field, etc.

[0059] Operational metadata refers to the description of the operational attributes of data, such as the data management department, the person responsible for data management, the data update frequency, etc.

[0060] In this solution, the source of technical metadata is still the factual source of model metadata. This part of metadata is not saved in the system. Business metadata and management metadata types are manually entered and stored in the system. This solution ensures the low-latency consistency of the retrieved technical metadata and the factual metadata source.

[0061] The central metadata retrieval interface dynamically assembles various metadata, and the assembled metadata is returned to the front end for rendering in a standard return format. Various metadata are organized in parallel and enter the front end through a preset channel.

[0062] The metadata involved in S1-S6 and the metadata management architecture of the factual metadata source, the metadata management architecture consists of a front end connected to the central metadata retrieval interface, an API Controller connected to the front end and serializing the metadata into the client request format, a Service Layer connected to the API Controller, and a metadata plug-in Manager connected to the Service Layer and used to obtain metadata.

[0063] The corresponding interface is used to pass the retrieval metadata. In some scenarios, too many fields will affect performance. The metadata plug-in Manage can load some fields of the required metadata. The specific code is as follows:

[0064]

[0065] The metadata plug-in Manager is connected to MySQL, Dor is and Hive, which are the sources of factual metadata. The dynamic assembly of various metadata in the central metadata retrieval interface also includes the factual metadata sources belonging to MySQL, Dor is and Hive.

[0066] A CDC module is set between the fact metadata source and the central metadata retrieval interface. The CDC module realizes the configuration change tracking of the databases of MySQL, Dor is and Hi ve fact metadata sources. To avoid data redundancy, automatic cleanup of data with a two-day retention period is started, and a change tracking function is provided to record the insertion, update and deletion operations of the tracked table.

[0067] The execution script to start change tracking for the database is as follows:

[0068] ALTER DATABASE AdventureWorks

[0069] SET CHANGE TRACK IN = ON;

[0070] To avoid data redundancy, the execution script for starting automatic cleanup of data with a two-day data retention period is as follows:

[0071] {AUTO_CLEANUP=ON,CHANGE_RETENT ION=2DAYS}

[0072] ALTER TABLE dbo.Product

[0073] ENABLE CHANGE TRACKI NG;

[0074] The change tracking function is used to record the insert, update, and delete operations of the tracked table. The change summary functions used are as follows:

[0075] CHANGETABLE(CHANGES) – returns tracking information for all changes made to a table since the specified version;

[0076] CHANGETABLE(VERSION) – returns the latest change tracking information for the specified row;

[0077] CHANGE_TRACKI NG_MIN_VALID_VERSION() - Returns the lowest valid version used to obtain change tracking information from the specified table when using the CHANGETABLE function;

[0078] CHANGE_TRACKI NG_CURRENT_VERSION - Get the version associated with the last committed transaction, which can be used the next time you enumerate changes using CHANGETABLE;

[0079] CHANGE_TRACKI NG_IS_COLUMN_I N_MASK—Used to interpret the SYS_CHANGE_COLUMNS values ​​returned by the CHANGETABLE{CHANGES...} function.

[0080] WITH CHANGE_TRACKING_CONTEXT—Used to specify the change context, such as the originator ID, when an application changes data.

[0081] The script for querying changes is as follows:

[0082] SELECT CT.Id,Name,XXX

[0083] CHANGE_TRACKI NG_I S_COLUMN_I N_MASK(

[0084] COLUMNPROPERTY(

[0085] OBJECT_ID('Product'),'Name','Co l umn I d')

[0086] CT SYS_CHANGE_COLUMNS

[0087] )AS Name I sChanged

[0088] CHANGE_TRACKI NG_I S_COLUMN_I N_MASK(

[0089] COLUMNPROPERTY(Object_ID('Product'),'Pr ice','Co l umn I d')

[0090] CT SYS_CHANGE_COLUMNS

[0091] )AS Name I sChanged

[0092] CT SYS_CHANGE_VERSION

[0093] CT SYS_CHANGE_OPERAT ION

[0094] CT SYS_CHANGE_COLUMNS

[0095] FROM Product AS P

[0096] RI GHT OUTER JOI N

[0097] ON PI d=CT.ID

[0098] It should be noted that the three values ​​of CT SYS_CHANGE_OPERAT ION are I, U, and D, and the corresponding DML operations are INSERT, UPDATE, and DELETE as described below;

[0099] SYS_CHANGE_COLUMNS is the code of the fields affected by DELETE.

[0100] SYS_CHANGE_VERSION indicates the field code affected by UPDATE and INSERT;

[0101] The CDC module synchronizes the fact metadata source to the message middleware in real time, obtains metadata changes in a timely manner by subscribing to messages, and maintains data consistency through exactly-once semantics.

[0102] The CDC module includes but is not limited to INSERT, UPDATE, and DELETE of fact metadata.

[0103] Among them, INSERT, UPDATE, and DELETE represent insert, update, and delete. Most of the modifications in fact metadata changes are a combination of insert, update, and delete.

[0104] The message middleware searches for the channel corresponding to the matching middleware according to the identifier of the source of the factual metadata, stores it, determines the path corresponding to the channel according to the channel, transmits the message to the queue corresponding to the channel, and performs retrieval after receiving the query request of the central metadata retrieval interface.

[0105] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0106] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A low-latency consistent metadata retrieval method, characterized in that: The following steps are involved: S1: The metadata is divided into technical metadata, business metadata, and management metadata. The business metadata is stored in the metadata management system DB by entering the metadata through a manual interface, and the management metadata is stored in the metadata management system DB by entering the metadata through a manual interface; S2: the front end retrieves metadata through a central metadata retrieval interface; S3: The central metadata retrieval interface finds the adapted metadata plug-in through the metadata plug-in management to obtain the corresponding technical metadata; S4: The central metadata retrieval interface obtains management metadata and business metadata by querying the metadata management system DB; S5: The central metadata retrieval interface summarizes the technical metadata, business metadata, and management metadata and returns them to the front end.

2. A low-latency consistent metadata retrieval method according to claim 1, characterized in that: The central metadata retrieval interface dynamically assembles various metadata, and the assembled metadata is returned to the front end for rendering in a standard return format. Various metadata are organized in parallel and enter the front end through a preset channel.

3. A low-latency consistent metadata retrieval method according to claim 1, characterized in that: The metadata involved in S1-S6 and the metadata management architecture of the factual metadata source are composed of a front end connected to the central metadata retrieval interface, an APIController connected to the front end and serializing the metadata into the client request format, a Service Layer connected to the API Controller, and a metadata plug-in Manager connected to the Service Layer and used to obtain metadata.

4. A low-latency consistent metadata retrieval method according to claim 3, characterized in that: The metadata plug-in Manager is connected to Mysql, Doris and Hive, which are the sources of factual metadata. The dynamic assembly of various metadata in the central metadata retrieval interface also includes the factual metadata sources belonging to Mysql, Doris and Hive.

5. The low-latency consistency metadata retrieval method according to claim 1, characterized in that: A CDC module is set between the fact metadata source and the central metadata retrieval interface. The CDC module implements configuration change tracking of the databases of the Mysql, Doris and Hive fact metadata sources. To avoid data redundancy, automatic cleanup with a data retention period of two days is started, and a change tracking function is provided to record the insertion, update and deletion operations of the tracked table.

6. A low-latency consistent metadata retrieval method according to claim 5, characterized in that: The CDC module synchronizes the fact metadata source to the message middleware in real time, obtains metadata changes in a timely manner by subscribing to messages, and maintains data consistency through exactly-once semantics.

7. A low-latency consistent metadata retrieval method according to claim 5, characterized in that: The CDC module includes but is not limited to INSERT, UPDATE, and DELETE of fact metadata.

8. A low-latency consistent metadata retrieval method according to claim 7, characterized in that: The message middleware searches for the channel corresponding to the matching middleware according to the identifier of the source of the factual metadata, stores it, determines the path corresponding to the channel according to the channel, transmits the message to the queue corresponding to the channel, and performs retrieval after receiving the query request of the central metadata retrieval interface.

Images